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ABSTRACT 

The management guidelines are divided into three 
-parts. Part I contains a number of questions vhich site staff ought 
to try to answer when establishing or while operating a 
computer-based education (CBE) site. Each question is followed by a 
brief commentary. Part II contains suggestions* for selection and 
training the staff of a site. Part III contains recommendations and 
suggestions about topics not previously introduced. The author 
suggests thax all administrative staff responsible for the (proposed) 
site read those sections of the guidelines which are relevant to 
their situation. The questions are arranged roughly from general to 
specific and, though they frequently refer to features and situations 
found on the PLATO IV system, most of the suggestions provided apply 
to other CBE systems which offer the author great flexibility in 
designing courseware. (Author) 
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Introdu ction 

The management guidelines are divided into three parts. 
Rart I contains a number of ^questions. Which site staff ought 
to try to answer when establishing or while operating a 
computer-based education LCB^l sit.e. Each question is:- 
followed by a brief commentary. Part II contains sugges- 
tions fob .selecting and training the-^staff of a site. In 
order to keep the commentary to the questions in part I 
brief, detailed ^discussions of .some topics have been placed 
in, part III of these-guidelines • Part HI also contains 
recommendations and suggestions about topics not introduced- 
in ^parts J and^ II. For succinctness, some terminology common 
to CBE is. used without .previously defining the terms. 

" The author suggests that a]Jl administrative staff 
responsible for the (proposed) site read those sections of 
the .guidelines which are relevant to their situation, return 
to part I to answer the questions for their project or 
organization, then compare their answers to those of other 
administrators. The commentary following most of the 
questions should help in formulating cJe;:;isions . 

A critical incident study was completed as a prerequi-r 
site for preparing these guidelines (Steinberg, 1977)/. 
References to the individual cases in, that study are given, 
in pointed brackets, "<X>". 

The questions in the following section are arranged 
roughly from general to specific. As can be seen from the 
table of contents, the second half of part I relates- only to 
sites wishing to use^ the PLATO system as an instructional — 
-medium. 
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In order to provide specific detailed advice and 
examples,, this document frequently refers to features and 
situations found on tH^ PLATO IV system (Lyman, 1977, Wood 
1975) . Nevertheless, most of the suggestions .provided apply ^ 
equally well to other-CBE systems whicb offer the author 
great flexibility in designing courseware.. 

The staff members of the Military Training. Centers (MTC.) 
Group and the PLATO Educational Evaluation and Research (PEER) 
Group, contributors to these guidelines, have gained much of 
their experience through working with CBE developments 
within -the military. In addition, the staff ha ve^ all 
had experience with civilian, academic, or Industrial CBE 
implementations* In. general, we have found classification 
in terms of site objectiviss, cognitive level taught, etc. 
more: useful than classification- by tnilitary and non- 
inilitary^ Therefore, situations specific to military users 
are noteST'throughout the guidelines rather than in a 
separate section. 



PART I-FUNDAMENTAL ISSUES IN PLANNING A CBE SITE 
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SITE OBJECTIVES 



To what agencies, offices, and organizations does the 
CBE group report? What objectives do these people-— 
have for CBE? 



This is a very important consideration for .the future 
of a ;site, add-it-ionaT funding, etc. One cannot simply read 
the charge or charter given to the CBE staff and assume the 
goals stated there are complete .or accurate.^ In fact, they 
have often" been found to be unrealistic or hastily-written. 
For example, one* reason> for implementing CBE niay be to- gain 
recognition and enhanced public relations. Although this 
goal wc:uld;^necessitate high publi-c visibility, the 
"official" objectives -might refer only to educational uses 
and purposes. 

In many cases it is imperative as- a first step to bring 
the organizational expectations for CBE into line with 
rer.lity, gently, so that no'ohe loses interest or respect 
for the abilities and^ opportunities of the CBE system. 



To what extent do the various administrators and 
agencies supervising the, site agree on the objectives? 



Disagreements on philosophies of education, orientation 
towards operational training vs. research, etc. have seriously 
hampered efforts at several sites (Himwich, 1977). Do not 
opt for a middle-of-the-road position just to satisfy people — 
it is often untenable. For example,, t.he ."operational 
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^ter-ching vs. research" issue is basic and pannot be resolved 
simply by alloca^tinft manpower* within a singl e project (e.g.;, 
two staff r.enbers for research, four staff members for 
i^nstruction) . The divide-and-appease strategy may work when 
several projects are available, but the constraints of 
research are nearly always in opposition to those for 
teaching;. 

Lastly, don't assume that because you have assembled 
everyone in a room, gotten their consensus, and even signed 
documents to that eff^ect;,. that your job Is" now over. While 
.such a step is important, it is generally taken early in a 
site's history,^ before;, impressions have developed. It it *— 
necessary to keep, in touch wtth everyone (especially non- 
project staff) to maintain their support and cooperation'. * 



Does the implementation .of CSE at the site satisfy a 
srngl e purpose^ (i.e., the fulfillment of a project 
commitment) or multiple purposes (i.e., general sup- 
port of its parent organization)? 



These guidelines, distinguish between single purpose and 
multiple purpose sites- and the suggesti^ons appropriate to 
each at sev^eral points later in the text. Typically a 
single-purpose or direct-support site has. a definite, 
limited staff engaged in fulfilling a research or dev.elop- 
nient contract of a fixed length and for a limit^ed audience. 
In contrast, a general-support site exists as a learning 
resource- It 'nust attract authors to write lessons, but can 
seldom pay them or exert much control over their product or 
process. This type of site acts as a utility, distributing 
resources and services to a roughly-defined group of users. 
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tI6w„ let^ us consider the an'swers to the question. 

Direc.t-support only . In this case the objecJtiy.es and 
management are .relatively straightforward-. CBE users have 
Qnly a ;sin?le objective and the site has only one way^it can- 
succeed. For example, a course director given two years, 
a staff of four and ten, terminals to see if CBE could be cost 
effective in an inventory control course would set up a 
direct- support site . 

Gen er:al>> suppor t only . A CBE site organized to provide 
general support tc. many small projects and to a dive" 'se set 
of users and purposes is ternied: a general-support site. For 
example, a department or school which has 20' terminals and 
hires some trainers, consultants, and classroom proctors so 
that its instructors can try out CPE and use it in their 
classes would set up a site of this kind. 

At a general-support site, it imperative to establish 
and examine the objectives of the site- carefully t CBE ^ 
^^rminals have sometimes become a solution in search of a 
problem.. 'Converting general ideas aboiit possible benefits 
of CDF into a half-dozen or fewer specific objectives for a 
site is an appropriate way to start. Reviewing these 
objectives every three or six months can help channel CBE 
resources into t.he. most promising developments. 

^Poth . If the answer is "both," relative priorities must 
bje established early to avoid, conflicts . General-support 
tasks can easily* fall to the project staff, excessively biir- 
di^ning them with unanticipated and unwanted jobs. 
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Obje ctiv es Ton a O ene ral-sug^ rt S ite 

,*fc\ Managers of direct-support sites may wish to read, but. * 
* :n6t answer, the questions below^ * 



Dp you already have clients? "Will the first task be*? 
to geriierate interest? 



Some people will tear the keyset out of your hands to get— 
to use GBE. They can sjee its potential and how they can 
use i\t.. They are i/irUng to spend the time to learn what 
they need^ to do' to apply- CBE in their v^ork. Other people 
are busy, cautio.ua, and skeptical. They Have seen tdo many 
educational innovations come and go. Often after two to 
four years they will accept and embrace CBE — after they have 
seen materials in their field teach their students, for 
example.. Short-term projects and innovations are trouble- 
some company. Deep-rooted commitment develop? slowly. 



What are -criteria for apportioning terminals, staff 
time, etc. between types of users, e.g., instructors 
vs. students, research vs. instruction, organizational 
use vs. private use (income tax, games),, and groups 
within one user type: chemistry students vs. biology 
students. 



Hjxampl e si tuation. An instructor or his department 
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reserves time for students to take lessons assigned for a 
class. After a few weeks, half or fewer of the students are 
using: ^t he/ CBF classroom at the reserved; time. Many have 
found' it more convenient to work i-ate at night during 
i)»1jcheduled t imes ,« others can, be presumed to be letting 
thel^r assignments "slip^" and there may be some dropouts, 
Rrbblem: the CBE classroom is reserved, but the terminals 
are, now under-utilized. V/Jho can use the classroom, under 
what conditions, and: how can scheduled students be 
protected?' 

Another example . The psychology department requests use 
of terminals, not for teaching or research, but for access 
to recreational lessons. They normally pay "volunteers" and 
wish to use GBE games as .a reward for participation in their 
«exper iments. Can they use CDF this way or does this create 
a double standard? Why can^t the history department 
schedule recreatdonal use for their students and staff? 



How focused and structured will ,the project be? Will 
it offer a service to all qualified to use CBF:? Can 
students not enrolled in a course use the computer? 
Can any staff member use it as s/he sees fit or will 
.only designated staff work towards limited goals use 
CBE? Will .QBE projects require .approval? 



wiil on-site programming consultation services be 
provided? Can a user expect to have programming done 
by a programmer? 
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If you wish to attract new users, or if you <can't 
provide much released time for potential authors, the 
presence of a friendly programmer may mean the difference 
between moderate and miniscule lesson development. In fact, 
depending on the characteristics of the expected: lesson 
developers, it may be unreajsonable to invest only in 
hardware and/or co.itiputer services without also including 
funding for support staf f <3B1 :Hote:^3B1 i 4 is the number 
of a case from the critical incident list (Steinberg^ 1 977,) • 



Wha.t amount of resources can be expended: towards 
. cheap, but--npn-objective- fulfilling uses? 



A typical situation; at a site trying to improve the 
training of electronics! technicians, the instructors from 
the medical group at the school find there are lessons 
available which they would like their students to work 
through. Other than good will^,, the medical usage won't hel"^p 
the evaluation of the electronics course. What dp you d'o? 

Because of the vast courseware available on the PLATO 
system, this problem arises frequen.tly. Furthermore, ^earXy 
rn a project., when few locally-written lessons are available, 
the terminals are generally under-utilized, making refusal 
difficult. If permission is given to use terminals, 
instructors begin to depend on the PLATO lessons and are 
indignant when they can no longer use them because the^time 
and terminals are reserved for other lessons and students 
<2A3. 1>. Acceding to such requests consumes more than 
terminal resources. Typically disk space for routers, 
courses, and comment files is needed. Because the "out- 
siders" are unfamiliar with operating the system, additional 
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tirne must' be spent setting up the routers and teaching them 
how to enter a ^class roster and monitor student progress. 

Much good is earned and often new and successful 

.projects get started from such Informal ^beginnings, but it 
also -happens that these efforts have drained .talent and 
resources from a struggling site. The key is to identify 
those services which can be provided with the lowest 
cpmsumptioh of scarce resources and which will generate 
good will in the most promising places. 

Qb^Tec ti s_ for a CB E Project. 7 

XThe following questions relate.d to project- goals 
should 'be asked of EAGfi potential fjeheral-suppor t user.) 



'Why IS CBF beinn; implemented? 



The .general answer to fehis^ question comes in two forms: 
'•We .think we can dp ^X* better if we have, CBE" or "We have 
sqme problems which we* think CBE might solve." Both are 
reasonable answers,, but the latter suggests some additional 
advantages. In order to jus;tify the use of CBE in the 
first case, one must find unambigupus measures of "better" 
and then appraise the value of the improvement. For example, 
how much is it worth to increase the final, test score of 
cpmmercial pilots from 80^ to 90?? Or how much is it worth 
:to decrease dropouts by iQ% and increase course attitudes to 
half a standard deviation above the school average? There 
is the also the prpblem that if CBE is declared l^etter, 
someone else is implicitly determined to be ppor.er. 

The questions abpve are relevant but have a different 
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perspective whenever CBE is^ introduced because sorneohe has 
identified problems: such as unacceptably low performance ori 
.af final test pr a high dropout rate related to poor student 
.attitudes. When QBE is introduced under these conditions, 
it :is probable that other methods have been tr^ied^ ^ari'd-" found 
insufficient;, hehce any solution is likely tqrbe -welcomed by 
all. Furthenmore, the introduction^ of QBE terminals is less^ 
likely to be interpreted as a threat or ari^ insult to^ the 
t"eaching. abilities of the current staff . 



In what domain do you expect CDF to show an ad.v.ahtage? 



Reduced /titn e^and co^t s - When: and if accomplished, this 
is one of the. most easily- justified reasons for implementing 
CDF. One, of the most difficult tasks in performing a compar-t 
ison of costs^^for XBE and other media is the detertnination 
of the costs, for the alternatives, The costs for developing 
and deii'vering materials; via conventional media have a 
tendency to .get "built into the system" and hence may be 
overlooked. For example, photocopying charges may get moved 
from the training budget to the administrative budget or 
-costs for transporting students to an: instructional center 
may come from the travel budget rather than the training 
budget. It is possible, however, that a .given CBE implemen- " 
tation may eliminate certain of these costs, thus enhancing 
its cost-effectiveness. To become aware- of such areas of 
savings,, one must diligently search out all cost items for 
the conventional instruction. Saving time Cand thus money) 
of.ten means shortened training arid' reduced staff. Unless 
the time and staff can be used elsewhere, there is no real 
saviTig^. 
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eB& costs are also difficult to calculate since afnortiza 
tion/periods for courseware developn^ent and hardware are nbt^^ 
.standardized. Furthermore, .when new deyelopment i-s occurring 
.it is difficult to accurately appprtipn the effort directed 
towards maintenance so that a true "operational?' coist can be 
determined. If a CBE site is new,, it is hard> to estimate how 
muchr oT the initial training and^ set-up costs: are one-time 
charges that should be treate<}- differently than other 
recurring costs. 

Because* the rneasurefherits of costs an s a y ing s* -a r e^ 

complicated, a special effort beginning at the inception of 
the project, should be made to gather ,^-recbrd^, arid^analyze 
cost data. 

Enhan ced- stu d ent . p erf ormance . Enhanced, performance 
incrudes broader student skills, higher proficiency levels, 
or higher cognitiv.e levels. Measuring an improvement in 
student perfornjance requires a vaird; measurement instrument, 
in many cases this might be a standardized test of achieve- 
ment* In others, it might be a briterioh such as the percent 
age of students^ passing the bar exam, the proportion being, 
admitted to a college, medical or graduate school, or the 
number of mechanics or realtors receiving certification on 
their first attempt. 'In cases where an accepted "yardstick"^ 
is not available,, a conside rabl e effort may be required to 
generate and^ validate appropriate measures.. 

Attitude s and, motiy/at ion. Improvement of attitude arid 
motivation are qften expected t*^ accompany innovations-. How.- 
.eyer,. unless poor attitudes and motivation haye been observisd 
and blamed for other problems, "happier" students, and staff 
\are not necessarily viewed by administrators as an asset 
worth increased costs. 

Quest ibnnaires and rating forms are the most popular 
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waysof measucing attitude, but tangible actions are probably 
more convincing- indicators (e.g., the fraction of students 
taking -another course in the subject, the.^nuniber-choosing 
a> carreer in the field, the attendaWce record or the voluntary 
dropout, :hate) . 

New . serv i ces . The opjportunity to ^take advantage of 
;new services not available previously is frequently cited 
as a reason to acquire CBE service; Eor example, it may 
be impossible to offer realistic simulations without a ChE 
facility. Although this reason is valddv it usually has: 
a more: fuhdamentai base in 'one- of the other reasons cited 
above,. e..g.., to enhance student perfprmance-by providing 
tsimulated) experience requiring higher cognitive skillsr 
such as analysis, synthesis, and decision-making. —"^ 

Resear^h > PLATO services provide 'tremendous opportun- 
ities for research* The availability of vast^:k^nds -^nd: . ^ 
quantities of student performance data allows finely-tuned 
educational research to be carried out. The flexibility of 
the computer and terminal allows investigations of human 
psychology to be quickly set up, impiemented , analyzed, and 
dismantled. Father than reworking hardware to adjust timing,, 
images, or control of the ex.neriment, the experimisnter can 
merely modify programming. The standard terminal p'eripherals 
:are sufficient for many experiipents and the terminal's 
ability to control other devices extends its flexibility. By 
putting all the directions on the computer, the experimenter 
can eliminate subtle differences in the directions given to 
various groups. Directions can be precisely controlled over 
a long „period of time despite changes in staff, etc. 

General Considera tion s. Many worthwhile goals cannot 
be measured immediately: (e;g. , ease of updating knowledge, 
success in non-classroom situations) . Nevertheless, they 
should not be eliminated from the list of reasons, for 
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acquiring CBE services.. 

K frequehtly-o'verlooked point is that CBE per se 
cannot fulfill the -objectives for its implementation listed 
above. That is, simply writing and^br using CBE. lessons 
will -not. automat icaliy; improve attitude, performance., 
retention,, or offer the pother advantages available. CBE 
services offer a medium through ^which such advantages may- 
bfe realized--not a guarantee that they will be simply 
'because CBE is being used. If students usirig/CBE lessons 
have shown higher retention, it is because^ teaching strat- 
'egi4s which provide high- retention haye been: used.. Because 
many of these strategies require a CBE system such as the 
PLATO sy.stem^ to handle their complexity,, people have erron— 
eausly attributed the effects to the system rather than to 
the ef fects_achievabie via the system. This misunderstanding, 
has. led some sites arid projects to "write some lessons arid 
measure how they are better"-ra path primed for failure. 

.Since cost-ef fectiveriessj^ialways seems to^ be an issue^ 
of foremost importance, we have prepared a list of uses 
for CBE which are potentially cost-effective. Naturally, 
many factors determine the actual cost-effectiveness of any 
endeavor, but the .uses suggested have shown or suggest a 
higher^than-average ppteritial. The list is found in part 
III'- of these guidelines. 



Howi will cost-effectiveness be measured? 



This" aiiestipn carinot be answered lantil the definition 
for cost-effectiveness is broken dowri and clarified. If 
the site or project must show cost-effectiveness, must all 
parts of it be cost-effective? That is, can some research 
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or service efforts be carried at a loss because of their 
overall beneficial nature-? For example, consider a site 
which' is nearly ''breaking even" ahd^ is also supplying free 
statistical and computational services to depar.tmental and 
agency istaff, Even if the users of the statistical packages 
cannot be made to. pay fo-r .the service, perhaps'the value of 
the service provided to them can be reckoned .and subtracted 
from the expenses of the site. It also may :be practical to 
show that some groups, or operations are cost-effective, 
though hot all are. 

^A^:secpnd consideration revolves^ >around- the time- period 
for which cost-effectiveness is calculated:. If one includes 
courseware developi^ient costs,, the amort.^zation .period is 
important. Aiso one must consider whether courseware devel- 
opment costs for CPE lessons should be measured against 
cpurseware development costs for traditional media.^^ If a 
new course has been written to incorporate CBE lessons, 
one must measure cost-ef f ectiveness: in terms ofH;he costs 
for replacing, ^the CBE materials ii;ith conventional instruction;. 

A third consideration is whether cost-effectiveness 
must be demonstrated while the terminals are under-utilized^ 
For example, consider a course which saves $lbO/student for 
100 students/year. If the CBE. terminals are in use 1/3 of 
the time for that course and: cost $20 , 000/year , one has tv/o 
ways to calculate cost-effectiveness. Assuming fewer terminal^ 
cannot service the class, one can ^pbint cut that costs 
exceed savings by 000/year. On the other hand, extrap- 
olation of the costs and savings to full utilization of atl 
terminals indicates a savings of *1d, 000/year . 



What features of the CBE system do you intend^ to use? 
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this question is .discussed more thoroughl:y> in/part III, 
"Why -Us.e -a .CEE System Like PLATO?" That discussion suggests 
-that, using only a\ single feature of the" PLATO system may not 
be ;j^ustif_ied: less sophisticated devices may; provide mpre 
^co St- effective cpmputatibn^ power or graphic terminals or 
large netv;6rks. Use of a system like the PLATQ system is 
especially justifiable when several such features are needed 
together and/or when the exact nature of the task cannot be 
pre-specified.. For 'example, if one kh6w:i the best. v;ay to 
teach bookkeeping, one may be able to choose a CBE system 
(or other medium) which just meets the needs of the training 
.ah8 provides no extra capabilities... 



;What is the nature of the project effort? 

^ This question is important fpr deciding the number -of 
terminals, choosing the staff,, hhd planning the overall 
goals for a project. 

Indirectly instructional . This would ihciude experi- 
menting with hardware and software; using. PLATO ^otefiles 
and communications netv/ork for surveys , etc. ; emproying. 
lessons arid packages for graphic, electronic, arid. :mechanieal 
design or ^statistical routines; or studying GBE as a computer 
science subtopic. These uses require few terminals and 
little, coordination; orily a small support staff would be 
rieeded: in most cases. 

Instr ucti onal rese arch. A research effort would require 
mbder<ate amburits of lesson design^ little effort devoted^ 
toward establishing arid operating a classroom, and high 
emphasis on collecting and analyzing data. At least one 
person with high programming proficiency would, be required.. 

« 
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OporatiolTal- teacJiinR/train ijip, . The major emphiisis hene 
would be toward efficient production; of high quality mater- 
ials. A curriculum designer and the lesson designers would' 
form ah" important part of the staff.. Liaison with students 
and instructors requires appropriate person?ility characterr; 
istics within the staff. The p.oals of low-maintenance, easily 
proctored lessons and standardized classroom .procedures dis- 
tinguish this from^ other types of use. 



What i5 the magnitude of the project .effort? What is 
the total number of hours of material to be prepared? 
What is the total quantity of human resourpes available? 



These are itppor'taiit ^qujf^stions Whoso answers must be 
compared to see if the resources match the task. In many 
cases proTject planners hav^^ failed to calculate the rate 
.of production required to -meet the stated goals. In other 
cases -*they have developed more materials than the si^te'.T 
.terminals can deliver. They failed- to calcuTate the Avail- 
able terminal hours per week and compare that to the class 
size <201.c, 2Dl;3i 2pl-^>- Directions for appropr^ ate 
planning calculations and obtaining estimates, of expe^^te.d- 
values are given in part HI of these guidelines. 



How important is it that the products (lessons, service 
routines, packages) from the project be used at other 
sites? 



Although widespread usage is olpten a good, objective 



23 



16 



indication;, of the general' value of a -series of Instructional 
materials,, it is difficult to fot:c.e. When: .the materials are 
Welfl-written and effective, users may flock around without 
encouragement from the developers.. However, taking steps to 
-guarantee that matfyrials will gain rapid acceptance in 
several institutions may be counterproductive. Generaliy, 
in order to promote acceptability>, the proposed courseware 
Is planned j outlined, and' critiqued by a committee of 
proposed users. Finding formats and stnategies which are 
widely acceptable may result in wringing the "life" out of 
.the instruction. On- the other hand, i'ghofi'ng committee 
recommendations may limit the number" of users; In summary, 
widespread use of lessons is a worthwhile goal which mLy 
sometimes best be pursued, indirectly. Efforts to produce 
high-quality lessons which are easy to use and> which are 
widely known through seminars, brochures, etc. may be more 
effective than efforts to create a product acceptable to all 
potential users and compatible with their Ideas. 

A frequently-ignored issue is that a project or site 
evaluation is necessary whether or not research questions 
are being answered. Also, the evaluation must be planned as 
the project is being initiated, ^hot as it nears the end 
<4A2.1, UA2'.3>.. Belpw are examples of issues to be negoti- 
ated and questions to be answered during the initiation of a 
project or site. 
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SITE/RROJF.Ct EVALUATION 

(A site manager reading this section shpuld.substitute 
the word' "site" for every occurrence of "project".) 



For the project to survive and thrive, who MUST ^be 
,pleased? Who else SKOIJLD be pleased, if possible? 
Tor each of them,, what will constitute a satisfactjory 
project? 



The general categories here are": the level above the 
project Csponsor, supervisor), the project level (colleague, 
instructor )'i and ,the level served by the project (client,, 
student") This question might be restated as "What"^ are the 
current problems of each CBF-related group?" and scrutiriizihg 
those problems^ which CBE might alleviate or exacerbate:* 

:Administrator * In order to fund CBE, a president or 
board of directors or sponsor might have to »be shown one or 
more of the f 61 lowing indications of "success": 

(a) cost savings, 

(b) total dropout/fail rate cut in half at a cost of 
lesis than $500 per student retained, 

(c) overt evidence of strong acceptance by instructors 
(CBE-related publications, requests for transfers, 
CBE time and/or terminals), probably accpmpanied by 
media coverage, 

(d) "remarkable" improvements^ in studea€- attitude , 
attendance, enrollment or performance* 

Instruct or. Most instructors would be satisfied if 
their total workloads were decreased or if their students 



showed heiphtened in.terest* Instructors would be happier if 
grading, test preparation, paperwork:, and,- remedial teachinR 
were reduced, giving them more time :for classwork.. They may 
fear losing their job to CBE, particularly when "cost savings" 
are mentiohed. Lastly, they want to feel that theV are an 
essential part of the instructional process. 

Students. Students, would be pleased to do "something 

-V * - . . ^ 

different.," spend less time on. "busy work," and work at 
their own rate. Some would like to learn to program and 
others would like to play pames.. They don't want to have to 
Tearh another skill (using the CBF. system) \in order to learn 
.the ^course material. Unfortunately, some may dislike- the 
thoroughness and attenti v.eness which are required in order 
to complete a CBE les;son--they can't sleep in ciass;^ \Students 
enjoy the ability to make mistakes privately,, the immediate 
feedback supplied by -CBF., and the :fact tlialr they control the 
speed of presentation. 



t Whose responsibility is it to see that project goals 
i are realized? 

I Experience has shown that even when sites ha.ve carefully 
^considered and chosen objectives eariy in the project, they 
-have sometimes failed' to see them: met because (a^) the objec- 
tives are nev^r discussed with and supported by thq project 
Staff, ^(b) because progress towards meeting long-term goals 
is seldom checked by management, or (c) because responsibility 
for meeting certain objectives is not assigned to anyone. 
•An example of the last problem is that at one site it was. 
generally agreed that "enlisting instructor support" for CBE 
was critical to the attainment of project success. The 
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executive manager 'Celt that the task would be done iby the 
manager, the manager felt that the the task could .best be 
done by his, staff., and the staff felt that liaison was the 
manager ^s role. The, circular logic was discovered when the* 
-traditional staf^^ exhibited resistance to viewing or using. 
CBE: lessons. Though the CRE project had been set up to 
solve a v/.idely-recognized training, problem, the traditional 
staff was unaware of the projeat's gpais. They perceived 
the CBE staff as an outside group who^ thought they could 
bring in, a .machine and, without working with the current 
i^eacb.in:& -staff, do a better job.. 



Does *tHe pro/ject have a definite end or re-evaluation 
point?; At what points in the project are evaluations 
of success/f ailsUre going to be made? 

Failure to realize th^^riearly every project does 'have 
potential ending points can be unfortunate. Furthermore, it 
is essential to realize that most go/no-gp- decisions will be 
made prior to the actual ending date or contract period . 
Thus the extra effort heeded to prepare an: interim report of 
successes and achievements may be a far more effective 
investment than effort devoted toward writing a final report 
once all the data has been collected <3B5.1>. 



If hardware delivery, courseware production, or student 
usage goals cannot be completely met, can a reasonable 
evaluation be made? 
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Althouph optimism is important,, it, is only pr.udent to 
consider and prepare for potential problems. In a sense;, 
th.ese guidelines, are a detailing of potential problems for a 
site 5r project; the section "Contingency Planning" in part 
ii'I of these guidelines lists specifically some predictable 
problems and options. A report edittrd by Sternberg ( 1977:) 
examines over 100 incidents in the evolution of^ PLATO project 
By learning about possible difficulties, a site manager can 
take steps to protect and stabilize a site. 



'VJill the evaluation assess the attainment of a pre- 
es^kblished goal or Invplve a comparison with an 
existing course- or program? 



this is a basic question that determines the character 
of the project and the evaiua^tioh Avdescr ipt ion. of the 
pros and cons for each type of evaluation^ is presented^ in a 
chapter in part III- Briefly,, it is: extremely difficult 
to. avoid comparisons v;ith other teaching metlia arid techniques 
even if the formal evaluation does not require iti 



What, dimensions will the evaluation^ examine? 



Because project objectives vary so widely, this section 
can provide only suggestions about what might be measured 
and a few warnings .about commbrily-overlboked items. The 
suggestions beiow should be regarded as a "shopping list" 
from^which to select measures important for the objectives 
of the project. 
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Ouaritity of use 

Contact hours (The PLATO system automatically 
record's hours/terminal, hours/ site , and 
hours/user, but it doesn^t distinguish 
between reoreational and inistnuctiohal 
lessons/) 

Number of students or users ^ 
Number of instructional staff involved 

Costr 

Production CAre trarninfr costs included in 

development costs? If so,- costs .will be 
rather high, in i^laH^^ Jf riot, an' appro- 
priate amortization rate must be ^chosen/) 

Delivery ('If development is going on simultaneously ,„ 

it is difficult to apportion costs between 
these two activities, V 
General (When considering costs, «add overhead of 
retirement, fringe benefits, space, etc. 
As rioted elsewhere, the mariner of report irig 
costs is, important. In one case the costs 
for developing PLATO materials was ILi times 
higher than for coriventional materials, but 
the number of users attracted to the PLATO 
materials was 9 times more,)^ 
Value of new services (Even, if no one will directly 

pay for using the CBE system to help do engineeririg- 
homework problems, end-of-course grade averaging, 
and statistical processing, the value of those 
services can be estimated.) 
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» IC the objectives of a project or site are instructional,* 
* sonie of the: following measurements may be relevant. * 

Student performance and attitudis 

Consider using these; scores on standard tests, 
class gradeS:, attitude ratings, pass/fail ratios, 
absenteeism, a^ttendence at voluntary sessions^ 
lesson completion ^times, test completion times, 
error fates in lessons, use of self-tests and 
voluntary review., extra practice problems done, 
retention or trainings, number who take follow-up 
courses, number in~* current course who enrplled 
based on -reodmrnehdati^oh b^y prior students. 

Quantity of Jesspn. production 

Hours, onrline averages .or replaced^ hours jhay he 
measured. (;A cbmpi ication : how can routers, 
computer-managed instruction (CMI) routines, or 
packages for circuit design, statistics , e^c. be 
incruded? they have no "completion time.") 
Non-terminal time savings (reduced homework 
reported by students, reduced test preparation and 
grading reported by staff, etc..) may be recorded. 

Quality of lessons 

At least five dtpfererit measures are possible: 
use by others (number of users, amount of use); 
colleague opinion (formally or informally assessed); 
student opinion (questionnaires, on-line comments, 
- the number who take follow-up cpurses); student 
interaction data (error rates, proportion and type 
of unanticipated responses, requests for -HELP-, 
fraction of questions: answered correctly on first 
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attempt); arid .the need for proctbring (number of 
students one proctor can handle);. 

'Note: Many of the suggested .evaluation variables were 
:taken from the October 18, 1972 PLATO Evaluation Note 
entitled "Some Evaluation. Variables for PLATO Authoris,^" by 
R.. Allen Avrier. 

« # « * it^it iMM « * * * * f ^ « * # * * * # « « i « « « « « « 

* If .students' performance and attitude variables are to * 

* be examrned^ the following questions are relevant; * 



what is^ the total flow of students? How. many times 
will class>s start- during the evaluation period? 



The flow rate is a critical element in. determining the 
structure of any evaluation. It imposes an* upper limit on 
the number of students who can be included in the evaluation 
Jecauise forimative evaluation during; iessoh development 
requires student "guinea pigs"^ to test the lessons, the rate 
at which validated lessons can be produced is dependent upon 
the frequency of new classes. Because evaluation plans may 
also need revision and refinement, courses which frequently 
have new classes forming are easier to evaluate. For exam-^ 
pie, a one-year evaluation period may be sufficient or even 
excessive in a military training situation where classes of 
10-20 students start every two weeks. In a college or 
university setting only two or three classes would start 
during this same period, .and a primary or secondary school 
environment would provide only, one opportunity to test the 
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students. No matter how many 'studeht.s are avai;iable^,. a one- 
shot evaluation, is extremely, hazardous. 

What is the total impact on students in terms of num- 
ber of hours on CBE or the total .fraction of training 
they receive from GBE? Is the impact signifrcant 
enough to produce .a measurable effect? 

Giyang half the students in a class, a four-hour CBE 
lesson rn the- middle of a 16-week ^cpurse is-unlikely ta 
produce a^^ measurable diTference on. the final test. The 
sen.s;itive and thorough data collection features available 
via^ CBE make many immediate measurements of effectiveness 
easy, but iorig^rterfn effjects' of fsmair or moderate amounts of 
CBE exposure may be difficult to detect. 

In order to perceive a measurable effect on icours g 
variables (dropout rate, course attitude,, etc.) 10% or more 
of the course may have to be taught via CBE lessons; Fortu- 
nately, individual chapters, modules:, or sec.tions of a course 
(;i-2 ?hours) can be successfully evaluajbed if one can* examine 
uhcontamihated "micro" variables. For ex ampler performance 
on. several test items covered by the CBE Instructiph and 
measured immediately after the instruction or within a^ week 
or so .m^ay .show an effect '(if there is one). However,, it is 
less likely that the same^ variable measured at the end of 
the semester (when other studying or sources of information 
are likely to supplement the st-ydent's knowledge) will sho\/ 
as sharp a difference between treated and untreated groups. 
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Can. you design a measure which is- fair, but which is 
sufficiently sensitive and specific so that student, 
compensation for good or bad instruction won't wash 
put di^f ferences? 



A test which asks questions related ;to examples or 
problems .given in a CBE lesson is unfair to students, not 
learning from QBE,., Dut questions which are fair can also be 
self-taught by extra? study. 

ir^non-GBE instr,uctfon for a to.pic is terrible,, CBE 
instructi^on is excellent, arid students (particularly highly- 
motivated ones) khd'w that they will be rated, graded, or 
selected oh the basis of their knowledge about the topic, 
their motivation may cause them to compensate :by studying; 
extra hard: differences between types of instruction may be 
erased. 

Can students and instructors be'^suf f iciently controlled 
so that students in, a npn-CBE section really get nonet 



Hysteria .about being denied CBE training has not reached 
the level which medical research must deal with, but the 
free-flow classrooms associated with college/university 
settings, combined with full review privileges may lead to 
adulterated samples when highly-motivated students feel they 
are missing out on^. something- Conversely, CBE-trained 
students who felt CBE training was incomplete have been 
known to study the notes and texts of students from control 
groups . 

33 



26 



•Will students be compared to existing standards? 



If existing standards -are used, the value of the CDE 
lessons rhay not be clear..- Because CBE lessons are likely 
to be recently constructed, they may te.ach ^l^opics which 
standardized tests do not cover yet; because they may use 
simul^ations and^ inquiry techniques to teach to a higher 
cognrtiv.e level than was possible previouslyv old forms of 
tests may nv t indicate the greater effectiveness of the 
CBE materials . 

If one creates hew tests,, how can performance on two 
dififerent tests be compared? The best answer to this 
dilemma is tg duplicate testing. Continue to use old test 
feorms for a while to show (hopefully) equally gooHI results 
rwith the CBE^materials. Simultaneously develop new. tests,, 
show, that they measure appropriate knowledge or skills 
( i..e..i validate the tests) then measure *the performance of 
students receiving both^ traditional and^ CBE* instruction. 
-It's a lot of work, but the only way out of the^dilemma. 



Are standardized end-of-c6ufse tests available? 



If so,, and if the tests in use are valid for the project 
ofe'jectiyes , you can save a great deal of time by not devel- 
oping, new tests,. 



Are any tests given upon entrance to the course? 
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If- entrance tests measure backpround and aptitude for 
learninp, the subject, you muy be able to use them to verify/ 
the validity of an evaluation. ^For example, when ^dividthp; 
classes for a comparative evaluation, -it's useful to show 
thnt both clnsses are equivalent. Such measures may also' be- 
useful for statlstrcal control or for assessment of aptitude- 
treatment intorac ttons. * 



Will the evaluation assess the project itself? 
the staff? :the training? The management? 



Who wi'll carry bat. ev.aI.uation :^ ipro ject staff or an 
outside group? 
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PROJECT PLAHHinG 



What proportion of the total :avail>able 'terminal 
hours/week are being set aside for projeat use? 



This question is designed to_prevent oversch^eduling of 
terminal resources.. In ordinary circumstances, one; finds 
that terminals are used about 30r40- hours per week. At 
open^use sites such ,as universtties an additional 20-30 
hours per week ape logged during night and weekend periods. 
Some interpretation of these numbers is needed. One can and 
should schedule more than 30-40 hours/terminal/week, but one 
should hot expect that actual terminal .use wiLl be 1001 of 
what is scheduled. Students who finish early,, students, who 
are absent, and class.es that, .don't use all available termi- 
nals contribute to a "shrinkage"' of the utilization fate.: 
Xhe 20-30 additional' hours^caririot ordinarily be scheduled 
because students may be unwilling to take the risks of lost 
data or uncertain computer availability.. 



What is the basic orientation of the staff in charge 
of the CBE site?'^ 



Three types seem to dominate: computer scientists, 
educational specialists, and departmental staff. They each 
- have strengths and weaknesses.. 

Computer scientists . They master the system quickly 
and they may be able to^perform maintenance, but spnie of 
them have a predilection for disassembling equipment and/pr 
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connecting add-ftional equipment, -either of which makeS: 
us.e for ordinary- purposes more difficult. 

Fdiicatipn _specialists . their abilities with instructional 
design, may be^ a significant help in producing, high quality 
Courseware;, but their concerns for instructional design may 
lead them to decide that only experienced designers shpuld 
be allowe;d to write lessons. In some cases their zeal for 
evaluation will produce so many surveys and questionnaires 
that potential users wiTll dwindle. ' 

pe pa r t m e n t a 1 ; s t a f f . VJhen a department acquires CBE 
terminals theVe is sdmetimes a tendency towards unreasonable 
possessiveness ; no matter how under-utiii2ed*The"""E"5rTO4^^ 
are, no one but students or staff from that department is 
allowed Jtb use them. 



At the end of the project,, who will own or control the 
equipment: the sponsor, the principal investigator, the 
department or group funded, or the ^parent organization? 



This is one item that should be answered by a careful 
reading of the contract rather than by requesting an oral or 
written opinion from an administrator. 



37 



30 



FINANCIAL CONSIDERATIONS 



Win the terminals and computer equipment be insured? 



Items should be inventoried so that insurance policies 
or other protection will apply to stolen or vandalized 
equipment. 



Will a department,, agency or group which uses CBE be 
rewarded in their budpet for being innovative or charged 
for using an experisi.ve resource to lighten their load? 



Both of the views above seem logical: a department 
-should not be penalized for taking advantage of CBE, but, 
conversely, they should be charged in some way if teaching 
burdens are lightened. Perhaps the only benefits are to the 
students. In any case, justifications for such administrative 
decd-si'bns are subject to criticism from either side and 
should thus be consid.ered in advance. 



Vfhd will own lesson copyrights and collect royalties? 



Funding Agency — It may want them^ to be free for all 
users, but it can't provide maintenance or 
updated, versions. 

Parent Institution — It provides office space, possibly 
terininals or more. 
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Department--It may- provide terminals, released time, 
and students. 

A,uthor--He may claim^ authoring was done in "free" or 
"research" time (in academic circles, this is 
simlTar to acting as a -textbook author).* 



Have you prepared a complete budget? 



The list belo.w includes some frequently-overlooked 
expenses incurred by_ CBF sites. 

Long-distance telephone calls (Troubleshooting 

terminal problems is normally attempted this way. 

The large distances between most CBF sites 
"increases normal telephone usage. ) 
Staff acquisition (Interview trips, etc.) 
Conferences (Since exchange of information^ is vdtal in 

this rapidly changing field, staff members must' 

travel to conferences-) 
Consultation (Experienced CPE staff are hard to find. 

A site may have to engage consultants to. provide 

certain skills, and services*) — 
New journals and library acquisitions 
Postage and shipping for repair parts and repaired 

terminals 
Above average copier usage 

Room preparation (Lighting, sound absorption, air 
compressors and pipe, extra electrical power) 
Furniture (Terminal carrels, cjiairs-, study tables) 
Printer supplies (Paper , ribbons or ink) 
Supplies for peripherals (^•icrof iche , audio disks, 

iilustratbr Tees, film, developing, tape recorders) 
See Francis (1976) for estimates. 
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.ENVIROHMEMT AND PHYSICAL PLANT 



^Where should the ^terminals be located? 



,Unfortunately;, some laboratory environments are too 
hot (over 80 degrees F)., too dusty, or contain corrosive 
atmospheres. Some sites have installed telephone lines to 
small classrooms, demonstration rooms, and offices so that 
a few terminals mounted on carts can be shared among infre- 
quent users. In lighter moment?, thiis is referred to as 
"terminal a la cant." 



.What Is^ the size of your proposed CBF classroom? 



Be sure to include room for carrels, communication 
equipment, audio units or other peripherall! air compressor 
for microfiche, microfiche library area, proctor ^location , 
and possibly a study area for off-line activities. 



What is the distance to the CBE classroom from other 
normally frequented areas (housing, food centers, other 
classrooms, or libraries)? 



Surprisingly, one of the strongest consistent dislikes 
college and university students have expressed about PLATO is 
the distance to the terminals. Several otherwise-promising 
miirltary uses were discarded when it was realized that the 
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time savinRS achieved^ by PLATO instrudtiori would be offset 
by. the time needed to get to and from the PLATO classroom • 
'On the other hand, usage statistics suggest that scattered,, 
tsmali ci^assrooms (6 :to 8 terminals) are often inefficient 
isn utilizing their terminals. 



What physical arrangements are planned: a learning, 
resource center.,, a media: center or CBE classroom? 
.Wli^l there :be separate locations f,or authors? 
Kill authors use terminals in a. separate room,, or 
have reserved terminals? Where will dembs^ be held? 
Will individuals have terminals in^their offices? 



Terminals in offices often .have low use.. An authoring/ 
demonstration; room^ may be advisable so that, students are not 
diisturbed by visitors. Install only a few terminals so that 
all are nearly always occupied and authors must overflo"w into 
the student classroom. Recauise it is more likely that authors, 
will use "jstudent" terminals than that students will use 
"author" terminals, total terminal utilization will be maxi- 
mized with this arrangement. A further benefit is that authors 
will be able to observe what's happening in the classroom. 



Will project staff membisrs have their own offices? 
Where in relationship^ to those of other staff? 



It is often valuable to give staff with part-time CBF, 
appointments a second office near the CBE installation in 
order to encourage them to devote a fair portion of their 
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time to CBE activities* is a mistake to establish a CRE 

office area away from the offices of subject matter experts 
and instructo^rs workinR in: related topics, this tends to 
promote polarization and to increase liaison problems. 



How many hours a day will terminal access be permitted? 
Who. wilX maintain a schedule? Who will proctor the 
terminals? How will access to student and^ author areas 
be controlled.? ' 



The answers depend on the type ^bf site and the type of 
use, but several points seem to find general agreement: (a) 
i(t is ineffdcieht to employ authors, as proctors, althouRh in 
smadl sites (fewer than .8 terminals), it may; be necessary. 
Good authors have a talent which is too rare and costly to 
warrant interrupting >their production with details which 
could be handled by a proctor heinp paid a lesser wa^e: (b) 
is inefficient to lock up the terminals after 5:00 p.m. if 
there are^ potential users for eveninp hours: (c) unattended, 
unprootored terminals quickly accumulate food and beveraf^.e 
debris and have become tarp.ets for vandalism in a few case3* 



Will study materials, tests, microfiche, etc. need to 
be checked out from a proctor or library? 



These needs are additional considerations which may 
in f luehce/.the location and layout of n PLATO site, .'vinco 
many sites haj/^e only microfiche needs, .several special 
arrangements have been found to be workat)ie. oome sites 
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which service only a limited user population (which needs^ 
only a ^^few mi.crofiche) seal the slide selector door from the 
inside so that the microfiche can be .permanently left in the 
terminal. =Wheh several different microfiche -are needed^ at 
one site, lessons may be designated 'to work only at certain* 
terminals. Some^ sites check out microfiche from- a^ nearby 
library. Both kinds .of sites have found that theft- 
.prevention procedures .are needed. 
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STAJF' SELECMOM 

^ i 

5t affin g a Di r ect-Suppo r t (P roje ct) S it e 



Wrll the staff be recruited from inside or outside the 
organizatioh , department,, course, etc.? Can new staff 
be hired, or must existing staff be rearranged? 



In ord^er to acquire staf:f experienced- l-n-^GBE^ one- - 
:g-eneraily needs^ to looK outside the organization i Haveyer,^ 
if the staff contains too large a, fraction of new people, 
.the project may be considered an outsider's venture. Some - 
existing staff members wi.th^ numerous, bfeoad-based contacts 
Cnot necessarily "old-timers") should be included for 
purposes of liaison., especially in a project whose goals 
include operational teaching and training <2C3. i| 2C3.2>. 



will the CBE project have a separate staff or will 
most or all of the staff hold other/previous department 
affiliations and supervisors? 



There is a potential problem "collecting" the released 
time promised by depar tments--especially when the fraction 
of released time is 25% or less. Also, if control of 
assignment, recommendations for promotions, etc. is retained 
by another department, there is an authority vaccuum for the 
director of the CBE project. Unless the "home" departments 
are strongly committed to CBE, staff members may be rewarded 
for ignoring their CBE duties in order to participate in 
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departmental committees, tasks, and functions. On the other 
hand-, if. it is necessary to fund, a GPF project with "soft" 
money.', it may Ibe possible to lure taieated staf f only by 
::givinR them status ahd^ a position with .an established, on- 
going department; 



- 'Wi4l authors .work full time ^br wii 1 they d^ivide their 
time with other teaching, pperatinn:, or research tasks? 

This topic is discussed in greater length in\part I.i, 
"Choosing a staff. " Briefly., full time may be too much for 
some positions, such as authoring, but as noted elsewhere, 
structured tasks such as teaching .tend to get more than their 
'prdportioriate amount of ef^fort compared to unstructured 
tasks such as authoring CDE lessons. <?.CH:.A\ ?SJ\.2j ;2C4. 3, 
2C5.r, 2C5.2, 2C5.3> 

For example., one half-time author who taught in higher 
education attempted to prepare lessons in the morning and 
teach classes in the aXternpon. He found, that his mornings 
were punctuated by students dropping in, meetings with 
course staff, and "rush" jobs from his teaching tasks: 
writing tests j grading lab books, etc. In contrast, few 
PLATO requirements were so major as to cause him to miss a 
lecture or not supervise a lab session. He summarized: 
"If you don't have; , a. recture. ready, or a test graded^ you^ll 
get 'people very upset. If the PLATO lesson isn^t finishedj 
you just give a lecture instead. No one gets upset. Ho 
one can say how long it should take to write a CBE lesson." 



Will authors work independently or as part of a team? 
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This question is discussed in part It as well as in 
Mahler-, Misselt,. Schell,, & Atdermah (-1976) . Briefly, team. 
} sjbr'ud.tures .are. advised when a rapid start-up is heeded or 
iwhehr it. is -not practical to teach, the subjec.t matter experts 
"tcl Become CBE authors <3C3.3, 3C4.1, 304 ^2^, 3Cn;3,. 3C4.4>.. 

Wibl everyone, who is; trained in CDE authoring be 
expected to become part of. the project? 



The discussion in part .J I suggests, that, .if possible, a" 
;hon,-pnejudicial escape optioh-be provided so that people 
who .are hot removed by- early ' screening procedures^ can^ later 
be mbyed: to^ and fro. other projects, tasks, and .assignmehts^ 
within and outside of the CBF. project. This requires that 
jbb descriptions and staff expectations be prepared, with 
such moves in mind. Frequent withiin-project re-assignmehts- 
have been: an author who had difficulties with TUTOR or' 
with instructional design becomes a subject matter expert; 
an author wi^th exceptional skills becomes ia reviewer and, 
later,, , an ihstrnc.tional designer; a programmer .wrth inherent 
instructional design skills becomes an author;, and colleague 
reviewers who have learned TUTOR become authors <2C3.1>. 



Will promotions change job descriptions and force 
staff to leave or upset the project organization? 



Particularly in the military services?, but to a degree 
in civilian employment, promotions may disrupt the orderly 
progress, of a project by forcing new leadership roles on 
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'the ^promoted group member or^by forcing -these workers to 
positions outside the project. The pQteritial for thi-s 
problem should be- an^ilyzed is part of staff selection and> 
contingency plans shbuld be prepared: <2C2..3, 3A3. 1>. 

St af f ihg ..a 'General -^Support.. Site (Volunteer Authors) . 

this kind of site is sdmetitnks found in conjunction 
wath a project (or direct-support)/ site and, often occurs in 
public: higjher education rhstitutionsi Typical examples; 
would, include unrversities or medical schools where the 
authors are professors or Instructors who wish to create 
courseware for their own courses'. They might v;ish^ to draw> 
upon the talents of a. programmer pool arid to consult with an 
iristnuctiorial designeri The degree to which each would 
learn CHE skills would Yary^^ but rnany would not wish to try 
to becqnj^e experts in yet another fields 



Who is available to be an author? What are the 
practical limits and What rewards are available? Must 
existing tasks be rearranged or may new staff be added? 



Some agencies have erroneously believed that merely 
prpyidinf^ access to CBE services is sufficient. There may 
be false economy in riot providing a large enough support 
staff to aid and direct appropriate use of the facilities 
<3B1.4, 3R1.5>. 



Can a university author count lessons he has prepared 
as "publications."? 
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Up to how the, answer has generally been "no," but this 
issue- may be negotiable arid feelings abtfut it have been 
changirig. Thejmairi stumbling -block has been the absence of 
^jurying" a "review" comparable that a publication 
in a :p,rofessi6rial journal would receive* With or without 
a 'substitute for -this "review"' ft is: worth trying to get 
courseware counted towards publicatipns since this issue 
could be pivd.tal . to a professor already pressured for tirne*: 
At least one project ^has hired resp^ected author ities. to 
criti^ their CBE :pr6duction and has used the critiques 
as support for their ar^guments to give piibHcation credit 
tor CBE courseware* * 
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■AUTHOR -TPA-PMING 



Where will ne,w authors be .trained^? 



If a site has be.eh^ able to hire staff members of suffi- 
cient background^ -and experience to train .the rest of the 
groug^ih the CEE language, ihstr^uctibnal design,, arid using, 
the CBE -system i_.the answer to this question is. obviously- 
"on-site.^' Because experienced staff ore rare, many sites 
are forced to rely upon outside inst^'uctors*. The advantages 
to training one^s staf f_of f-site seem to srightiy outweigh- 
on-site training. 

On-^slte. If the trainers^ can; bi§ brought :to the site^ 
costs: are typically lower because fewer people must travel 
and additional staff can be trained or can consult with 
experienced users for only the cost of their time. Though 
the advantages of being trained by several different trainers 
Is lost,, it may be compensated^ for by a longer training ? 
period (made possible by lowered costs) . 

Off-site . Training at a large CBE center provides 
trainees with much broader experiences^ than they could receive 
'while staying at home. Important experiences Include:, 
visiting a functioning classroom ^ meeting other authprs and" 
^subject matter experts in the same/fieid, arranging for 
reviewers and users for future materials, and viewing or 
using optional devices available such as terminal peripherals. 

If many staff are to receive CBF training, but only a 
few terminals are to be installed at the new site, there may 
be an advantage to obtaininc training at a large QBE center 
where terminals can "^be reserved fpr training classes. 
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Can the ^taff get credit for the CBE training? 



It, is importarit for staff in the military service, as 
well as^raany primary, secondary^, and community coHege 
staff members to receive. Written^ cerltificates documenting the 
satisfactory .c.omp.letipn: of training. 



How can the project staff be^ protected^ from the: nbrrnal 
jealousy associated with special projects and advanced 
equipment? - 



Many oL the keys to preventing jealousy and- maintain- 
ihg cooperation have been mentioned previously: avoid 
hoarding equipment and terminal time, include part-^time as 
well as fullrtime staff, indicate with words and/actions the 
important role of non-project staff 'to the overall project, 
demonstrate regularly and fervently .the common goals, being 
.addressed by both project and npri-project staffs. 

The need for liaison to promote good will between CBE 
and non-CBE groups is nearly always- underestimat^ed- Tlie 
intrpductibn of a new technology which^ is not familiar to 
all the staff is rather threatening; The possibility of 
getting left out or left behind is clear. Efforts should be 
made from the beginning of any project to orient and intro- ' 
dtice staff to the CBE jsys tern. The construction of the CBE 
classroom should make it easy for non-project staff to 
observe what is happening and »4se the terminals themselves 
without feeling they are intruding <2A1.1, 2A1.2, 2A1.3i 
2A1.M, 2A2.2, 2A2.3,, 2A2*i|>. 
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* The remaininc questions in part I of these guidelines * 
'* hefer to situations, in which the CPE system is: to be * 

* used ^primarily, for instruction • * 

COURSE .bESCRIPTIOH AND ISSUES 

Throughout the next sectj.on it is important that the 
vconcept of a "course" is clear. We will use it to mean- a 
presentation of mui;tipie concepts during a period of a month 
or more. Single-concept, short term training presentations 
will be refered to as "mini-courses." 



ts^^course lock step, group paced,, or self paced^? 
Will this format continue to be^ used? 



Lock step . This format allows for very easy scheduling 
of lessons and very high terminal utilization rates. In 
higher education environments, however, students will use 
unscheduled times- and terminal locations, if available. 
This tendency lowers the utilization rate during scheduled 
times, but does not affect overall usage rates. Courseware 
in a lock-step format is easy to introduce into a course: 
the instructor simply informs the students that they should 
report to the CBE classroom rather than the lecture hall. 
Because .most students will .be taking the same set of lessons 
during a class period, computer storage (ECS) is efficiently 
used, and if the lessons are well-polished, little proctor- 
ing is heeded. Although, by definition, no time can be saved 
with lock-step usage, the intrinsic self-pacing of CPE will 
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normally force the instructor to make some- provision for 
slower students to finish during a ^period: outside normal 
class^ hours. It may also be necessary to proyide supple- 
mentary- materials for those^'who finish early; 

Group paced , Struc ture d, study; "Structured study is a 
situation in which all students in a class spend the same 
,fixed amount of time studying. This situatipn may be brought 
about by^limited terminal access time^. or :by the limited 
training day fjound in some mnatafy tTai'ning environments. 
In .a structuredrrstudy situation CBE strategies involving 
forward branching or pre-testing to skip sections of lessons 
and thus accelerate the progress; of bright or experienced, 
students are found to be self-defeating. They tend to put 
inordinate pressures on slow students because the self- 
paced , individualizing nature of CBE separates students in 
terms of total time to master the topic. Hence a lesson 
that is judged or pr6ven„ to be efficient under self-paced 
conditions may be i napp ropriate and even inefficient under • 
group-paced conditions. It is especially true that certain 
strategies "normally" used in CBE lessons will increase 
lesson production time without saving student time. This 
occurs because the rate of the slowest student determines 
the rate of the group. 

In both lock-step and group-paced formats, students 
finishing early can create problems. They must be released 
from; class or given recreational lessons or lessons on 
optional topics. (Giving them lessons used later in the 
course would only put them farther ahead). Slow students 
may ;have to retake a lesson in an atmosphiere of shuffling 
chairs and/or "pinball alley" (i.e. , while other students 
are loudly enjoying recreational lessons). Ac* the slow 

students struggle to finish, they realize that the progress 

if 

of the whole class is being held up until they finish. In 
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such -a situation both, fast and slow students save time by 
"exchanginR answers" to .the questions in the lesson. 

Croup-pacing is. often used when the nature of parts of 
the Instruction Is such that. s:erf-p"acing is' not possifrre, -e;-g 
demonstrations, close laboratory supervision, etc- It is 
often seen as a compromise betw.een self-pacing and lock-step. 
Because of ^the existence of the lock-step phases in the 
training, the number of instructors is generally the same 
as that Tor lock-step training;. TCS demands on PLATO are 
only slightly larger than that for lock-rstep. 

Despite the aboye problems , group-pacing can reduce the 
training oeViod required while adapting to varying .stiident 
populations. The latter is an important attribute for the 
.military services whose students may vary in experience with^ 
the time of year or the existence of a draft or mpbiiization 
effort. 

Group pacedj Free stud y. In this situation CBE could 
be used freeiy outside class, but the class wouid still be 
group-paced. Unforfcunateiy , our experience does not include 
any examples of this training format. The most typical 
.free-study environment, public education, supplies few, if 
any, payoffs for early completion. Because-presentations 
and demonstrations typically require the orderly completion 
of prerequisite materials,, this, format is of limited utility 
Self pac^d . Selfrpacing is ideal ly matched to a CBE ^ 
environment, but it requires stand-alone equipment setups 
or additional instructor/proctors in cases where laboratory 
practice accompanies the course. Because students in a 
self-paced environment often use CBE at random times and 
places, an instructor is typically unavailable. In most 
PLATO CBF. situations, this presents few problems, since 
notes between students and their instructors are easily 
written. 
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Wi'thout firm guidelines, students tend to allow self- 
paced' coursework to be displaced by assignments with definite 
deadlines. :Self-pacing provides both the problem and the 
solution here. In some institutions, such courses can- simply 
be finished the next term.. Alternatively ,' The instructor 
can set up intermediate milestones with deadlines that must 
be met by self-paced students. Because self-pacing suggests 
that computer-generated or computer-selected quiz and: test 
questions are needed (for test integrity) , production of a 
self-paced course may take longer than for a group-paced or 
lock-step course; However, administrative problems associated 
with nemediatiion, absences, etc. can be nearly eliminated. 



Wha.t are the roles of instructor in each of the 
training formats? 



Lock_step. Because this is the familiar, conventional 
teaching format, it will be used as a basis for comparison 
but will not be discussed explicitly. 

r-r ou p p ac ed , Str uc Wed^tudy . An experienced military 
curriculum^^deVgloper argues convincingly that this mode of 
operation is the worst possible for instructors. Because 
the slowest student sets the rate, the instructor cannot sit 
down to review and carefully teach the slow student — that 
would mean the group would move at a still-slower rate. All 
the instructor can do is to prod the student, or sit at the 
-terminal and give him the answers so the class can proceed. 

Self pace d . In this mode, the instructor can stimulate 
fast students and help slow students. While the comput'fer 
carries the burden for most information presentation and 
testing.. The computer's dominant role may bring out the 
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best in dedicated staff, the worst- in lazy staff ("If CEE is 
.s,6 good,, let it do all the Work"), Some instructors take a 
while to adjust to the idea that they are no longer the 
"niain attraction" knowing all'the information and deliverrng 
it from hrgh atop the lecture platform. In fact the role^ of 
an instruator is much mpre similar to- that, of a prfvate 
tutor. Once instructors- adopt this vi^ew of tfierr role, they 
generally find, it rather rewarding. 



I:s the course presishtly organ ized^ into moduies., 
chapters, and: lessons? 



These may provide convenient divisions for allocatihg 
assignments, scheduling production:, and organizing changes 
to a course. 



Are course objectives available? Are old tests and 
exams available? Lesson plans? / 



If new staff members are to develop course segments, such 
documents and information are extremely valuable for 
providing directions. Developers should be cautioned, however 
that following old lessons or lecture notes too closely can 
be far too limiting. Old tests can be-used to indicate if 
the same level of mastery is being achieved as well as 
perhaps providing some provocative questions for the CBE 
materials. It is worth some effort to retrieve such tests. 
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What ^media and strategies are used in current course? 



^ Film or video cassettes, filmloops, a u dio tapes . Often 
the ef feet iyeness%f -exist inR materials aan be increased by 
teaming them, up with CBE. For example*, a filmloop may 
contain, points oyerloofced by students. A computer-composed 
test may focus attention and ^produce greater mastery with 
little cost.^ 

Lab exercise s . In many academic situations, students 
cbtnplaih- because lab work seemingly has little relationship 
to^ classroom topics* This problem frequently occurs, because 
labs>must be planned weeks in advance so that equipmeni and 
supplies can be fresh arid ready. Variations ir? the class 
progress means that the' lab and lecture get out of synchrori- 
izatiori. potentral solution is to tie a short CBE lesson 
fco eactPl^b^in order to relate it to topics in the i 
ma-instream of study. 

Pre-iab simulations can familiarize students with the 
objectives, apparatus, and procedures associated with an 
exercise. This solves the problem of students who don*t 
read the lab directions before the lab, speeds equipment 
SPtrup thus allowing longer exercises to be completed, and 
reduces damage to equipment because of misunderstanding and. 
haste. 

V/hen the purpose oF^'a lab is to collect, analyze, and 
report data, CBF lessons can 'reduce the instructor's lab 
book grading time by accepting data, checking calculations 
arid analyses, and performing grading. 

brill . A community college math instructor reported 
that he once spent a lot of class time drilling students. 
He now can use class time to present concepts and he lectures 
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only once or twice a week while letting xhe computer super- 
visee the drills. In other cases, complex algorithms for 
determining when a. student should review missed items have 
beeht easily managed by the computer. 



What are the outstanding strengths and weaknesses of 
the course? 



An analysis of strengths and weaknesses should: ppint 
out the parts of the course which^ should be changed, the 
parts which should be retained, arid may suggest a direction 
to start when looking Tor solutions to course problems. 

If a course is known for its fine case studies, perhaps 
new case studies which are individually interactive could 
be added. Cri the other hand, if the course is known for 
bavinp; fine lectorers, they should not be replaced. Instead 
.consider developing a series of lessons to grade homework 
problems, provide prerequisites,, or deliver supplementary/ ^ 
enrichment materials. 



What kinds and amounts of homework are assigned? 



CBF.rassisted homework is potentially a way to upgrade 
performance, but it will not save money directly because 
hojnework time is cost-free in most environments. 



Will CPE lessons be used during class time or will they 
be considered like homework--to be done outside of class? 



50 



^Depending^ on the way CDE lessons are assigned, .students 
may treat CBE-taught topics differently. For example, 
st'uclents _wh6 are used to out-of^^LCjass. v^prk being practice 
exercises for what is covered in. class will treat 6ut-df- 
class^^CBE assignments similarly: students who do not nor- 
mally^ complete homework assignments are likely, to ignore C3E 
l^essons.. Howe.ver,. if most students hormally perform^ out-of- 
class assignments, heW^ topics and' additional material may be 
taught via CBE without using class time <3F3..'<>.. 

^.Will the addijtix)n of CDE expand the existing course? 



aGRE deyeloRers, sometime forget that adding adjuhct^ 
simulations, practice exercises, computor-aided problem 
solving,, mastery drills, etc;, will expand the amount of 
student time requiried to complete the course. Although this 
can sometimes be offset by providing labor-saving routines 
for statistical analysis, etc., the overall impact on the 
course length should be examined. 



" What is the student . population for this course — homogen- 
eous or diverse? Does the course have prerequisites? 
Is there a need for remediation? 



6ut-of-class remediation.-, for a handful of students may 
be a cost-effective, technique for reducing the completion 
time of a group-paced class or for reducing the dropout rate 



is the drop-put rate large and/or problematic? 

The aFswer depends on -the situation. In a public edu- 
cation institution, the dropout rate is probably irrelevant 
unless, it/is embarrassihRly large or unless aepartmehtal 
iluhdihg i^s dependent on the number of students, ooiripleting 
ithe course. In^military and iridustrial training inist itutioris 
andi in privatelyrsupported colleges ^ dropouts are typically 
expensive- CBE materials may bj^^.part of a potential cure « 
.for a dropout problem. 



Is the course taught to a. ievel of mastery and/br are 
grades assigned? 



CBE is ideally suited for teaching to a mastery level, 
but some' institutions require^ some sort of score,, grade or 
other disorimihatipn device. 

The decision to teach to a .mastery level requires more 
different forms of tests than are typically used in a 
-non-mastery situation since re-testing to show mastery is 
necessary. 

It should be noted that some students prefer NOT to 
achieve mastery of the material and find it frustrating . 
when they cannot relax and take a "gentleman's C" for the 
course.. Whatever the medium used, the introduction of 
mastery requirements eliminates traditional grading. Essen- 
tially, everybody gets the same~grade! 
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Will CBE lessons be phased in as. they are completed, 
or will- the course be suddenly cohverte:dt 



tddy research designs and the practicalities of forma,- - 
tiy.e evaluation and operational teaching prescribe different 
answers-. For "tidy" research, the full complement of CBE 
lessons; should, be implemented-at once (to all students: or to 
harr o/ a^ randdmLyVselected, group of :students). From the 
point of view of a department or instructor .trying ^tpt upgrade 
the training or solve teaching .problems or ,a= curriculum 
developer Wishing to tune- the courseware to the students^ 
the fayqned approach, is more likely to he to make a subj.ecr 
tive. evaluation of the value, of new materials and implement 
ilmedTately all those Which seem- compTeteTy debugged; <;2D1i5>i- 
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Who will decide what goes on to the C|3E system? What 
criteria will be used? 



As- indicated prey iously , this question is of critical 
-iirvPortance. instructors- or ihsti^tutcions^ distant from the 

developers are expected' tp- use the materials^ this Is a 
logical point for requesting their input. The criteria for 
choosing topics for CBE d'eyelopment depend^ on the project 
ob j^ectlyes j modified ^by the considerations of what CDE can 
dp especially well.. Kh .much^ experience as possibie: should 
.be employed when jpakihg these decisions. This is one of the 
times during a project when ah outside consultant niay be 
^.needed i - ' -™ ^ ^ 

Will the lessons written be considered to "belong" to 
a- course or to an instructor^developer? Will future, 
instructors be required to use some, any., or all GDE. 
lessons written by this organization? 



In a higher education environment, if the lessons are 
to be written for a specific course to be used by current or 
subsequent instructors, it may be important that potential 
users be involved at all stages. Requesting reviews from 
nbn-CBE instructors is often a good first .step. Failure to 
involve others may result in^ unused^lessons when other 
instructors teach the course. In a military or primary- 
secondary public education facility, instructors have loss 
choice about teaching materials and the problem outlined 
here occurs less frequently <3F1 . 1 , 3F1 .2, 3F1 .3, 3F1.il, 
3F2.2, 3F2.3>. 

6i 



Wild lessons be written so they can be used several 
different courses? 

On the surface, this path seems prudent because it 
allows deyelppment costs to be amortized more quickly. But 
there are some hidden pitfalls- Normally the backgrounds of 
stucients in the different courses will vary considerably. 

he igroup of students will be bored or the other strained 
\uniess Herculean efforts at individualization and ability 
tracking are made» T^ese^ efforts, of course,, boost develop- 
ment time. 

A, better approach is to write lessons for higher ability 
students, test and polish it for theth. Next , have at few 
students from another course use the lesson. Note their 
prxoblems, and modify a copy of the lesson to meet the differ- 
ing prerequisites of the other users. In many cases less- 
prepared students will point out difficulties which- should 
be repaired in all versions of the lessons. Nevertheless. 
It is so much simpler to write a lesson; for a single, harrow 
audience that it is worth the slight additional computer 
storage and moid i-fi cat ion time needed for multiple versions^ 
Furthermore, the author will not waste; any time installing 
extra help and remediation which are not truly needed. 
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THE STUDENT .POPULATIOM 



How are students selected for the ^cpurse? Are any 
'predictive tests administered? 



Knowledge about the students is impprtarit since many 
design decisions are based on the ..breadth and depth, of student 
abilities;. Listed prerequisites are often out of date and^ 
should not be accepted withouj: examination of current class 
makeup and/or future need.s. In technical and^ vocational 
courses, reading level arid niath skill^^^ should be cfiecked if 
possible. On the other hand:, doh>t spend time administering 
or retrieving test data (e.g., SAT scores) unless the use 
for them is clearly established. 



Is mot i vat ion a problem for students? 



If so, improved motivation may be an objective to be 
achieved via CBE. However, do not expect improvement 
merely because of the change of medium. Few improvements 
are automatic. One must use strategies to enhance the 
relevancy or increase student involvement, for example. 



What problems do students have during and after this ^ 
course? 



The effect of CBE can be more than proportionate to the 
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.fraction of time spent usinp it if bottlenecks can be ailev- 
iated and hurdles can be lowered, this Is also a fine way 
' to- enhance motivation^ 



Is terminal use to be scheduled or free flowing? 



since CBE is inherently seir-paced^ it is convenient 
if students can use terminals freely^ Thus they .can .skip 
classes when they are ahead and they oan find vacant terminals 
^wheh they are behind. In any case, some convenient uhsched- 
Mil-ed timie should be provided; for slower students.. This 
brings up an interesting semantic problem. The term ''slow 
stiicient" has become a euphemism for "poor student." Another 
term :must be fb,und to describe students in a self-paced 
mastery situation where "slowness" carries a much smaller 
negative connotation. 



Do students need additional rewards for achieving high 
performance? 

\ 

For example, some sites allow students" to play gamiss as 
a reward. It is probably as motivating (and certainly 
cheaper) to install a publicly-available "record book" for 
displaying the score of high achievers within instructional 
lessons. One successful format incorporates programming 
to eliminate the recording of embarra*?singly low scores and 
to cause the premature obsolence of,, say, the sixth through 
tenth best scores. The latter feature deletes some old 
scores so that achieving a "recordable" score does not 
become impossible . 
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How pressured: are the students for time?^ for grades? 

CBE will not be popular if students already :pressed 
for time perceive that CBE. assignme^nts take longer than- 
assignments for conventional media. Similarly, the use of 
CBE-given grades (or grades from any mastery-type training>) 
must be carefully considered for students who are especially 
anxious abqut grades (e.g., pre-med students). It is gen- 
erally inappropriate to "curve" scores from mastery -exercises 
in^ order to rprovide "grades:." Another problem with; grades 
based on performance in a CBE lesson Is that some lessons 
can be repeated by a student. Which grade should be recorded 
f.he first? the last? The .average of ail attempts? 
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:dfveloping a curriculum 



Are you preparing a curriculum, not just lessbn-s? 



Although some reasonable uses of CBE involve the 
creation of a single lesson not terriblx closely related to 
other topics being taught,- one usually' wishes to integrate 
the: CBE lessons with- each other and. with materials in other 
media. However, because many users start out .thinking small 
(especially at a generaL-support site);;,, it frequently happens, 
that a course eATefttually contains a collection o,f lessons 
which cannot be transformed into a curriculum.^ 

it is folly to develop a lesson, withpijt adequately 
planning for its integration into an educational context. 
If you are not planning to modify a whole course, hut rather 
are creating a few -lessons^ to -jpnesent some dif f icult-to- 
teach topics, the concept of a mini-course may be the best 
approach. A minircourse is an integrated cluster of on-line 
and off-line materials which teaches a single, limi.ted-range 
topic. The critical features of *such a cluster are: (a) 
well-defined,, but iimited prerequisites (b) consistent 
terminology, objectives,, and perspective and (c) a definite 
ending point (from which another mini-course can be launched). 

Creating lessons without a master plan can be ineffi- 
cient. For example, one developer explored the use of the 
CBF system writing sever aU.kinds and styles of lessons. He 
introduced the CBE system , ^the keyboard , and checked the 
student's math skills in each. As he developed clusters of 
lessons, he found that the students were being re- introduced 
to the CDF system (and having their math skills tested) 
again and again. Furthermore, several lessons just would 
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not "fit'* into a cluster. ^ventual3y all lessons had jblteir 
. introductions remcveH^ .and some lessons were divided into 
several parts s6: they could: fit into clusters. 



Will the lessons be used for mainline instruction or 
as an adjunct to other instruction? 



Mainline lessons provide an easier format to justify 
costs and. shbv^ savings, but they require more care and te?.t- 
ing during creation and irtpleme^ntation (because they are 
the sole source of information about a topic), if sufficient 
time is available, one path which is often succe.^sfuX is* to 
use the lessons in an adjunct mode inrtially until any prob- 
lems are eliminated (or a skeoticnl audience is won over) and 
then us^ them as mainline lessons replacing conventional 
materials. This cohverrlon from "adjunct" to "mainline" 
should he phased in beginninj^ as soon as possible because 
student standards for each type of usage .are likely to be 
different and the elimination the previous mainline^ 
matf>f ials may reveal some oversif,hts. 



Are you modifying an existinpr curriculum, dr creating 
a ne;/ one? 



Most people realize that unless they modify their 
curriculum somewhat, they won^t be able to make good use of 
*their CPF system; fewer realize that if their modifications 
are too ambitions, they may not be able to handle the- twin 
tasks' of curriculum development and masterin^j a new medium. 
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The decision abdut where to strike ^ balance depends on the 
experience of the p.roup with respect to curriculum, develop- 
riejxt and*^ CDF instractipna] desi gn as well as the manpower 
available. 

Because of the vast possibilities and the great number 
of choices available from a CHF-based ctir^culufn, sites With 
moderate curriculum desipn experience ^in limited r'ledia) have 
become ^Ired and have fallen short of their goals. Further- 
more, some kinds of evaluation are virtually* impossible if a 
major revision is mad^l si,multahoous;T y with a :change in media* 
^ Although writing an^ entirely new course is a very diffi- 
cult task, the result may no:t be so thf€*ateninn to current 
instructors, furthermore,, one may be able to use the CP-E 
system to avoid capital costs associated with labs or 
demonstration devices. 



Vill media other than Cr^F be used? Who win bo 
responsible for deciding what media to use for each 
topic and for insuring, that the media are interfaced? 



"^here are sip^ni^lcant problems with teachinf^ larno 
portions of a course only via cnr lessons; cnp resources pre 
expensi;ve, they are inappropriate for some uses, and they 
have unheeded power for some applications. *'ixint^, media is 
more efficient, but requires someone to intenrat;e fTv-^m the 
bep.inninc^ all materials written on a topic <?P'I.1>. 



What offline materials or .paper eopi^s of CRF lessons 
will bo made available? on what conditions? 
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One of the greatest shoftconings of ^PLATO GPE materials 
.produced so far is that they, attempt, .unsubcessf iilly , to be 
totally stahd-^aione • Although "providing manuals, worksheets, 
study t^uides,- outlines, and summaries to accompany C^F 
lessons is very sensible, it is jjQldom done, As^ a. result 
the -cost-effectiveness of some applications may be reduced; 
^Ror example, students who want to have something to study 
after .they have left the terminal are forced to take notes. 
Hotel-taking is ah .effective study„.strategy for some kinds 
^6f learriihg. TSut- cheaper alternatives than copying from 
a ;CBE terminal are available. Forcing students .to take 
-nd.tes" from the, terminal may change .a lessba which saves 
tihie (and: money) into a lesson which takes extra time. 

Instructors may fear that students will sicip the CHF 
lesson if they can pick up a summary of what they must Iparn 
rf a summary sheet can adequately express all the concepts 
which must be learned, there is ho reason that the CBF 
lessoh should be used. If the CB^ lesson is truly necessary, 
but the students are l?:zy, the instructor might wish to make 
summaries available only after the student has successfully^ 
completed the lesson or only several days^ before major tests 
<^F1 .P> . 



vill published lessons (written at other sites) 
comprise a significant amount of the courseware used? 



Yes . It is a major task to interface the topics and 
courses taupHt at one's own site with lessons selected from 
curricula. Written elsewhere. 'Someone should be given respon 
bility for gathering lists of available materials and aiding 
instructors in the se:iection of lessons uased on more than. 
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their ti-tles.. As in textbook selection, a problem sometimes 
occurs because materials which arc optimized for students may 
seen unstimulatlris and unappealing to ah instructor <2n2v5>. 
Suggestion: include several students oh the lesson selection 
committee. 

Modification of lessons written at other sites often 
entails a great deal of work unless the changes involve only - 
.the addition or deletion of displays and' text • ^ Documentation 
IS frequently lacking and there may -be prpblems* in finding 
the author and receiving permission to modify his work>. 

.No. Unless terminal delivery Is phased, the decision 
not to use materials written elsewhere will result in :a 
long period of under-iitiiized terfoinals. Furthermore, the 
relatively large development costs associated with ORE 
:rnay threaten cost^ef fectiveness if some existing lessons « 
cannot be employed. An insightful comment ^by a well-known 
educator unfortunately applies to many CBE developers: 
"Researchers and educators frequently demonstrate a strong 
resiistance to the use of someone else's Irinovation. It has 
been said that if there were a Nobel prize for educational, 
^research? we would have to nominate an entire generation 
of researchers for their co-discovery of the..wheel," 
(Molnar, 1971). 



Are lessons to be v;ritten simultaneously for all parts 
of the course or will you start from one end working 
toward the other? Which end will you start from? 



All parts ^^I^2^^^£^ly • This may be necessary if 
sever^al authors with different sub-specialties are employed 
Spreading the effort may make testing more difficult and 
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may require more effort to assure thajt obj;ectiyes and 
preronn i.s it es match: up* 

Begin ning at the epd^ When large, consecutive portions, 
of a course are to, be rewritten to include CBE instruction, 
it is frequently advantageous to begin lesson writinR and: 
implementation ?t the end of the course. This intuittvelyr 
urontf order Has the^ following advantages:, 

1. '^he first materials seen by a student (those most 
likely to turn him on or off to CBE) are those written: 
when the authors ?hd staf? rare most- experienced, in; CBF 
development. 

?. The process follows the order of a formal task 
analysis- if mid-way through the dev.elopment , one 
finds students unable to understand the material, one 
can easily add prerequisite topics because the 
development oT prerequisite topics is always "down the 
road" in terms of time. If a part of the task analysis 
must be postponed, development can nevertheless 
start immediately . 

?• If the course is berng converted from lock-step 
to self-paced as the CBF materials are introduced (a 
common occurrence) , students can move through the 
conventional materials together, then finish the 
self-paced part of the course at^their own rate. If 
the course were rewritten beginning at the st^rt, 
students usin^ the self-paced CBX-.jnate rials would have 
to catch up or wait for the whole class to begin the 
conventional lock-step section of the course. Any 
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time savings realized from self-pacing aV)d CBE lessons 
can be taken advantage of,- even during^ preXiminary 
Testing; phases, 

^. If for some reason th,e development effort falls short 
of its goals in terms of quantity of lessons produced, 
it is. less Tilcely that the. evaluation , plan will be 
completely invalidated or impaired,. As noted above, 
actual time savings can be measured, not estimated; 
CEE-influenced attitudes will be "fresh^;^ and^ the 
h'igher-level cognitive skills (qften^ taught near the 
end of the course) for Which CBF can often show cost- 
effectivehess will be present in the completed portion 
of the course. 
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I.F.SSOM fYP»^? 



WiTl tests i as well as lessons, be proRrammed for CBE 
usage? 



Some sites have used -CRF almost exclusively for 
testinp >^ecause of the capability for computer-Renerated 
test, items, easy re^cordinp, of test item statistics (e.g.., 
reliability,, difficulty level, dis^cr imination index),, and^ 
the opportunity for feedback' (Guerra , McOaniel, Kaufman 
Eulier, 1977). Other instructors have felt more comfortable^ 
using CHK , lessons, but keeping their testsr in: a conventional 
fomat. f^tiVl other users, faced with an evaluation plan 
calling, for a comparison of^students taught via GBE lessons 
with students taurtht conventionally, have realized they must 
test conventionally in order not to Rive unfair advantage to 
students ramiliar with the CEF system. 



What problems with test security exist? 



To avoid embarrassment, test security should be checked 
thoroughly before iteTis are entered on-line. Procedures for 
checking student identification should he planned and built- 
into tests • Allowance for system failures during testing 
must be made. 
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LESSON STANDARDS' 



Will all lessons produced 6h-si.te have to meet a set 
of standards? 



In- i\ehef al the irlea of "maintaining standards)* is a 
sound, pri>ctical one." It may be difficult,, however,, to 
impose such standards om professionals using tha terminals 
^at a general-5uppprt site.. Fortunately, the ne'fed for strohR 
standards Is greatest In a direct-support project si-tuatioh. 
the-- question of creativity: vs.i. control cannot yet be answered 
easily and objectively. An affirmative answer to the abq\j.e 
qu.ery naturally raises the issue of what standards will be 
set, ,who will enforce them, and how the enforcement will be 
implemented. Previous attempts to impose standards by reRU- 
lating: storage space for new'lessons have quickly raised 
author s^ ire. In other cases, the "enforcer" had a lower pay 
or rank than the authors and programmer and hence was ignored 
by them. 

Lt is overly-optimistic to expect ne.w staff to prepare 
polished lessons in their first attempt. Phased-in .standards 
are likely to be more app opriate-. ^or example, the first 
lesson mlf,ht be written without the imposition of any fornal 
standards. When the second' lesson is written, a copy of 
the standards could be provided, along with a justification 
for using them. As later x] essons are prepared, the use of 
standards would be encouraged and enforced. If necessary, 
the initial lessons written would be brought within the 
guidelines. Drawing up the standards with the aid of the 
author staff promotes motivation and understanding of the 
nee.d and use of lesson standards. 
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What kinds of .standards might be imposed? What purpose 
^ ^does each -^serj/<l7 Vi ^^^^^^a::-^^ 



* 

Docum enta tion * ^n appropriate level of documentation' 
protects the investment ^of the sponsor, agency, or department 
by "insuring th^at another user wishing to^ update,, correct, 
or modify the programming can do so in -an .efficient , reliable 
manner. It also insures that ;tfie author liimsetf Will hot 
waste time trying to interpret programs: he wrote long ago; 

w^tan dards fo rj;c a, CPU^ and.PAEM. These are abbrevia- 
tions for lesson size .and the demands which an operating les- 
son: places on the computer. If the demands are excess! yev 
the lessons will be unusable, ineffective, or will reduce the 
lesspn access for others at a site.. Standards for usage of 
computer resources such as thesje insure that the reputation 
of the site, authors,, and system wili be protected and that 
students trying to learn from the lessons will not be annoyed 
or impeded. 

Lesson format . Several aspects of a lesson's format 
might be subject to standards. If strategies inappropriate 
to CDE are used, cost-effectiveness v;ill become a problem 
an^ expensive development time will be wasted-. For example, 
lecture-like lessons are sure "loser^s" yet are commonly 
prepared by some authors. The presence of a title page, 
statement of objectives, and author name on each lesson can 
provide a cohesiveness among lessons, which student? find 
comfortable and efficient. The inclusion of restart points 
protects a student from system failures and allows him to 
leave the system and return without re-doing part of the 
lesson. By including data "cbllection standards, a general 




supRprt .stte can insure that authors who use site resources 
also generate infbrfnation :to;;pr6vide a basis or justification 
X6r^tR§/ continued support of the site. Data bbllection: 
s.tandards can also prevent errors in lessons from going 
undetected""and plaguing subsequent groups of studentSi 

Some sites include a requirement for a criterion* 
referenced test at the end of each lessorii Properly construe 
tedy such a test will allow a student to get "credit?* for a 
iessoh; only after mastery of the material has been dempn- 
s.tra^ed^ Without such a test, some, students feel that 
cqmpietion of the iesson means that further study of the 
topic is unnecessary. Authors of succeeding: lessons nay 
find that students are unable to understand those lessons 
because of inadequate mastery of the prerequisites. 

Most sites include spme sort of subject ntatter and 
editorial standards as well in order to assure that all 
facts are accurately stated and all grammatical, semantic, 
spelling,, and typographical errors removed.* 



If lessons from other sites are used, will -they have^ 
to meet the same criteria? Who will modify and 
upgrade them? 
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MAMAGi.tlG- FORMATIVE DEVELOPMENT (EVALUATJNG LESSONS) 



what criteria wi-l.I be used to determine when a lesson 
is fihtshed. or validated?- 



Although, few lessons, are ever truly "finished we use 
that word here to indicate that, a lesson nay be used by a 
class of casual iyrproc.tpred students. Some combination of 
student useClnvolving both -the total number of students 
and the number of testing/revision cycles) and colieague 
review should be established (and adjusted with experience) 
:s6 that lessons dp hot reach students prematurisly . 



How will lessons be evaluated? How often will they be 
evaluated? Will they be examined after they are 
finished or validated? 

Evaluations during both formative development and as 
part of a summative development are warranted. At some 
sites nationally-known experts in a field are brought in to 
review lessons which the site staff consider finished. 

The aspects of a lesson which are normally evaluated 
include: the programming, the instructional design, the 
conformance to site, standards, and the subject matter accuracy 
Ln many cases the evaluata^<^n is broadened to include student- 
performance results, student and instructor comments, and 
data concerning the quantity of usage. For a more detailed 
discussion of the aspects of formative development re"vlews, 
see Francis, Goldstein, and Call-Himwich (1975) and Call- 
Himwich (1976) . 
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Ar:e student "giJih^a^ p for testing lessons readily 
available? Whose respprisibirity is it to provide them? 
What rewards can they be offered for their cooperation? 



A group of eager students, possessing about the- right 
backf^round for the material and a readiness to try out lesso:n . 
fragment s,, can be ah enormously valuable asset for a: stjbe. 
The absence of such^ a. population allows authors to Vreate too 
much material before testing it with students* Since authors 
tend to do this under any circumstances:, one must present 
them with many easy opportunities t. btaih student :feedback. 

If sufficient terminal time is available, it may be 
;possible to "pay" studen^ts .bv giving them time Tn^recreational 
lessons. The amount of time they earn is proportional to the 
number and value of their comments . 



Are lesson objectives available? Will they be written? 
Who will write them and when will they be written? 



There are advantages to choosing a course for which the 
objectives have already been prepaired. It is easy to be 
deceived, however, unless one obtains copies of the objectives 
and^ reviews them carefully: the written objectives are often 
outdated and do not reflect the content and emphasis of the 
course. In many cases they are so general that they have 
little value. Written .objectives also tend to record only 
those topics which can be explicitly taught and tested. 
Looking at such too-narrow objectives may lirrLt the potential 
applications of CBE. Far example, the goal of having a ^ 
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student' ask questions*,, plan an invostiRatign , and carry it 
out may; be an unstated and unmet goal which CBE exercises 
could p.royidet. Similarly, the objective of training for 
discrimination of sounds hiay have been deleted because 
appropriate hardware was hot availabTe prior to the Installa- 
tions of the CBE system. 



Who is responsible for verifying that -the objectrVes are 
appropriate, in terms of content and level of knowledge? 



Are objectives, nepded? 



This question arises frequently, especially among staff 
who have had unpleasant experiences with writing and using 
objectives. Former classroom teachers may have been forced 
to use objectives by administrative fiat and may harbor a 
grudge against objectives because they -seem useless. 

Furthermore, rt is true that learning objectives are 
sometimes so bbv-dous they need not be stated. For example^ 
the need for students to solve a particular hype of problen 
nay be well recognized (e.g., two-dimensional, f rictionless , 
collision problems or non-disproportionat ion , oxidation- 
T_eduction equation balancing).- Tn these cases the objective 
is to be able to solve any problem of that type and tho test 
is :t6 give the student a problem and let him solve it without 
aid. A more formal, written objective is not needed because 
any subject matter expert seeing the lesson topic could 
immediately determine the material be ing taught. 
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TyjoicaTly, the lesson developer is not so lucky as to 
.have such "tidy" topics; to teach. Instead, the depth of the 
instruction (Are exceptions to the rule to he included?) and 
-the^ breadth^ oT insi^ruction (Where does this lesspri: end?) tpust be 
determined as part of the lesson development process. Pecause 
a CBE lesson will probably interact with more students, than a 
.classroom instructor,, the CRE developer can and should spend 
more time in the planning and objective writing than the 
lecturer. A well-written set of lesson objectives^an 
prbylide the developer with guidance and act as a touchstone 
sor the author knows what is: relevant to the lesson and what 
is superfluous, when the lesson is done, .and how to determine 
if \the lesson is adequate. 
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ESSENTIALS-SKILLS 

When considering what staff to hire, an adrninistratpr 
ty.pically begins -by forming job descriptions , then: looking 
fqr-rpeople Who match the job descriptrpns . Because pf the 
limited availability of staff with. CBE-related experience, 
we suggest considering the skills needed Tor the project 
rather than roles or job titles to ^be filled. Below we have 
listed ten skills typically needed within a CBE development 
staff. In a small projeat, two or three people may have to 
?provi'de all the skills listed; even in a large project, 
several roles are generally combined. Discause skills are 
present to varying degrees in different staff members , . 
premature combining of skills intp a job description should 
be avoided:. Analyze the contribution of each candidate in 
terms of which project needs he could serve and v;hat 
flexibility his talents would offer. 

' A d m in i s t r a b i v e / m ana g e r i^a L. This skill is needed* to 
coordinate the project, maintain standards, and expedite- 
noh-clerical details, this i^s a general project support 
role-- to smooth the path and monitor progress. It entails 
less decision-makinn than the skill of leadership, discljssed" 
n e X t:. 

tfJ^lT^^^P* "^his important skill provides direction, 
give.s rewards or , prods, sets a pace or an example, makes 
decisions, and initiates changes. 

Curriculum coordijiat ion . This skill is necessary to 
insure that the curricula or modules produced are well- 
intep>rated packages. It requires the broadest subject matter 
experience. Although this skill requires much coordinating-, 
it is not necessarily the manager's function. 

Subject-matte r e xpertise. This skill is employed when 
preparing objectives, providing content, writing text, choos- 
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tnp examples and non-exampt es , Hescribing typical misconceprr 
tions., pointinR :out exceptions to ^•uies, etc. ' '^>' 

Instructor ^experience. This skill is necessary for 
providing insight about the students'* backgrounds, problems, 
motivations, futures and typical complaijaJtsu 

I n s t r u c t i o n a i_ de sj. gii > This often o-yerlooked or under- 
estimated skill provides knowledge-about: the .process of 
.planning^ designing, implementing, and revising instructional, 
materials. 

E V al u a t i o h / 1 e s; t i ng:> These two skills each probably 
deserve categories of their 6wn,, but itheir relatedness ;me,ahs. 
.that they are often possessed by a single person- These 
skills serve to pr.bmote , implernent, and aid formati ve and 
summative evaluation practices ♦ Depending on the project 
objectives, the preparation of a research design and an 
arialvsis of riesults may also be needed. 

CBE/ljnguage expertj,se. This skill is needed in order 
to program lessons and advase in the efficient uses of the 
computer. It is also needed to explain and exploit the 
various communications features, the courseware design 

consicleratibns, and the system characteristics associated 

It 

with the CRF. system. Experience on other computer systems, 
particularly other CBE systems, is a frequently-needed 
compromise in staff selection which may broaden the total 
base of experience enjoyed by the site* 

Editorial skills. These skills enable a site to "polish 
the final product, to ''debug" the non-programming problems, 
to sharpen definitions and explanations, to correct grammar 
and spelling, and to insure consistency and smoothness 
through the instructional materials. 

Gener al support. The following skills ore needed to 
support the efforts of the primary CRE development effort: 
the skills of on-site hardware repairmen, proctors, secretar- 
ies, clerks, photographers and illustrators. 
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A very careful assessment of the skill needs and avail- 
able talent can take ad\^antage of the special backgrounds of 
the staff. Thus rather than considering that- one heeds a 
staff of a manager, secretary, five authors, two programmers 
and ean <evaluator, and that one should find people who best 
meet those descriptions, a wise- administrator can choose 
people whose experiences are-^'cpmp^lementar y even if the 
candidates • quaLificatiohq do not exactly match typical job 
descf iptiohs i A flexible policy in find ing personnel might 
allow: an administrator to choose a staff composed of: a 
subject matter expert iSMK) with cubriaulum: development, 
experience in another field; a .programmer with some testing 
expef tehee; a current course instructor who has subject- 
matter experience; an evaluator who has worked in instruc- 
tional design; a subject matter expert with computer science 
courses , but no teaching e'xperience: a CBE author with a 
' pbby related to the subject, matter ; a secretary ^who was, a 
Journalism major for a year ; a manager who is an experienced 
course instructor and has written or rev.Lsed many of the 
course materials; an SMF with some courses in learning and, 
instruction as well as an artistic flair; and an SME who was 
a technical writer in industry. 

We emphasize flexibility in staff selection because the 
most consistent finding of a study of PLAfO courseware devel 
opment procedures in military and civilian environments 
(Mahler, et al., 197.^) was that courseware groups changed 
their organization one or more times during their compara- 
tively short (two to three year) project lifetimes. The 
changes in staff roles caused by reorganization can be some- 
what minimized by emphasizing skills over positions. Skills 
will change rather gradually, and provide continuity during 
periods of reorganization . 
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staff reorganizations caused by weaknesses of the 
members aref sometimes simplified if skills rather than job 
titles are emphasized. If a staf-f member is unable to 
perform satisfactorily and discharge- and' replacement is, not 
feasible, reassignment of duties'may be the answer. '' For 
example, an author havih,'; trouble mastering the CPR language, 
might, be assigned only SME responsibility. Alternatively,, 
th^t author might be used as a proctor or lesson, editor, 
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SKILL COMBINATIONS 

Hhen choosing project staff from available candidate 

pools, one is forced to look for individuals who can provide 

'seyehai of the ten talents and skills listed above. One must 

a;lso< be prepared* to make compromises in filling posi*tions. and 

consider the flexibility of individuals shpuld their role or 

the project change. Suggested below are several common skill 

combinations which result in high product ivity. 

/the leader of a project pften has the additiona;! respohsi. 

b'iiities of curr iculum coordination ^ project 'management , or 

both. It is, however, important to see tftese as different, 

bui related:, roTes, An overworked project leader can, it 

necessary, assign these responsibilities to others while 

* 

st:i-l?l maintaining his authority and position. 

Assumption of the leadership ^and evaluation roles by 
a single person is- most common in research projects. Further 
more, in several rather successful ''.oursewrre development 
^projects, the leader acted as instructional designer for the 
group. We would suggest ^this combination be employed in 
those instances where an instructional designer has exhibited 
managerial skills or vice-versa* 

the curriculum coordination is often performed by an 
especially experienced subject matter' expert. Experience and 
breadth of knowledge are more important than depth, however. 

Instructor experience is generally cccompanied by sub- 
ject matter expertise. Mowever , mi? itary training instruc- 
tors are not so likely to be true subject matter experts. 
The lesson plan and training materials are often sufficiently 
speciTic so as to allow persons with narrow subject matter* 
expertise to be '?ffective as military classro.om instructors, 
but they may not be able to perform more general roles such 
as a reviewer for a CBF development site. These instructors 
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must be sup^plemented- by •'genuine" subject matter experts; 
they should definitely not be expected to supervise the 
curriculum development. 
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TEAM^ AUTHORING 

One of the decisions that will affect staff selection 
is the ^organizational structure of the staff,; At a general 
support site^ all the author iTig- may be done independently,^ 
but most project sites adopt some sort of "team approach. 
The concept of an authoring team varies from, site to sfte:, 
at least three differeht^ things are meant by a "team." 

lu In the loosest sense,, any group of several authors 
working coward the same set of goals and helping 
each other may be said to comprise a teanri. Hpwev^er, 
unless they' share responsibiO tties for programming, 
instructional design, etc., we classify them as 
independent authors rather than as members of a team 
<3C3-1>- 

2\ At the next level of tec.j association*,^ a single 
staff member makes the" decisions about "the desi;gn , 
content , and progra.mming , but "subcontracts" site 
support staff to furnish the content and/or the_ 
programming. ^ ^ ---^ 



3., The strictest team approach divides the tasks of* 
designing , furn ishing content , and*/ proper amming so 
that no single person makes deci'sions about all — 
throe of these areas* This approach may be compared 
to an assembly line where once a lesson has completed 
one stage of develx .ment, the control and responsi- 
bility ppsses to the developers oflthe next stage. 
In practice, loops built into the process give 
development groups a "second. chance" to see and 
modify the lesson. 
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The first approach requires the least coordination, and 
allows, the most -flexibxli^ for staff to enter or leave the 
project. However, because few authors possess all of the 
ten or so skills- needed for effective CBE development, there 
Is, the potential danger C^r creating an uneven, inconsistent 
set of lessons. 

For that reason, many sites starting, out with the first 
approach deiscribed above modify it to become more like the 
second approach. Groups of two to four staff members are 
'pl:ad.ed together so that the strengths and weaknesses in the 
yar-ious' ski^lls can, be: compensated fpr;. In general, the 
result is enhanced produatiy ity and /greater uniformity; In 
;some cases, authors who have been succesisf ully using t;he 
independent approach are allowed to continue in that mode. 

Management nee^s for the small-group teams are somewhat 
greater because of the additional effort to organize and 
cbor^dinate the teams. Generally, however ^ it is the 
m,anager's decision to assume a larger admini"s"trattve burden 
in order to achieve greater control of the project <3B2.1, 
3C3^3^^36^^.1, 3Cn.2>. 

In a team authoring, structure the productivity of 
individual members is more difficult to measure and the loss 
of one staff member affects the whole team. On the other 
hand, because the team members need. not possess so broad a 
range of skills, they are more easily replaced. 

Our experience with the strictest form of a team approach 
indicates that for authors familiar with other authoring modes, 
the lack of identification with a lesson seems to offer 
fewer internal rewards. However, from an administrator's 
point of view this reduction of ego involvement may be a 
desireable outcome (Francis, 1976, p. 32). The mo5^/c freouent 
implementation of the "sub-contract" form of team authoring 
occurs when subject matter experts plan and design a lesson, 



but delegate thej^ proghamminp responsibil^itie:s to someone 
<;ei'se " ^ ' i 

Evaiuatl<)n of the efficiency and effectiveness of eaph 
of these ajuthpring approaches is confounded by effects of 
the quality of .the .authors and the objectives of the site. 
cHighly-skilled authors tend to prefer and perform best in ah 
ihdepend^eht authoring mode. Generally., when this mode has 
be^ri tried and discarded, it was because the authors* inexpert 
iehce precluded their writing lessons with sufficient quality 
or in sufficient quantit^t to meet project, goals. Reduced 
author independence has typically been viewed as a. technique 
for increasing tbe quality of the average lesson and/or 
increasing the production rate. 

A, rather extensive discussion of authors, teams, and 
the dynamics of authoring, structures found on PLATO has been 
prepared by Mah.ler et al • (1976). 
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STAFF SELECTION GUIDELINES 

'in this section, we offer some comments concerning the 
selection of key staff members, the roles of director, ^' 
Instructional designer, and. author are discussed , and brief 
comments are made about severa^l other ppsitidns. 

Dineotor 

As is typicatly the case, the choice of director is 
crrtical to— the success of '*the Tpro ject . Therefore this 
fpositipn: is, discussed in the greatest detail* 

In general the fbirowlng have been found to be 
^true: (a) a directorship -typicariy cannot be handled part 
"time when the staff numbers three or more,^ (b) the director 
must have sufficient time available to be able to watch 
students use CBE lessons, (c) the director .must^^be open- 
minded enough to ignore preconceptions and intuitions gained^ 
in other media or with other student ^populations , and (d) the 
personal interests and background of the director xust be 
matched with site goals "(,e.'g* , don't hire an instructional 
designer for a hardware project anu vice-versa) <?6't.i, 2B1.2 
3A1.3>. 

A frequently-overlooked problem is^ the question of the 
•^distance" of the director or other higher administrator from 
the project. The project director must be far enough away 
from the day-to-day problems of authoring so that these 
matters don't obscure the long range view. Some directors 
have tried to add directorship responsibilities vo their full 
time author duties <3Ar. 1, 3A;.2>. At the other extreme, 
some directors whose jobs entailed non-COF duties have been 
frustrated when they are circumvented in the decision-making 
proces? because they had too. little time'to familiarize 
themselves with .the current problems and p.otential solutions. 
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When a project-oriented site is to be started, some 


*. 




addit-ibnal^ insights can be offered. It is most important to 






obtain the director as early as ppssible--as .much as 6-12 


• 




months' before the- arrival-of the full complement "^of staff and 


i 
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^terminals. This is especially, important if the director is 
^notjfamiliar with instructional development in CBE. An 
".earry start" has given many directors a chance to gain 






Experience and make a few. mistakes without incurring large 




\ 


penalties. If the director cannot be acquired early, he 


\- - 




should' be available at least at the beginning -of the project; 






those projects which had no. leader or a known-to-bertemporary 


i 




leader have tended to founder until the director was appoint- 






ed <2B1.1>.. In order to be sure that the project Objectives 
are well-understood, it is highly desirable that the project 
director participate in the goal specification phase of the 
proj^ect. 

At a general-support site,, the director need, not 
exhibit the leadership, curriculum development, or subject 






matter skills that his counterpart in a direct-support site 
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must. J^owever, both share a neeH for managerial, .CBE , 
instructional design, and media selection expertise <2B2.1, 


• 




2B2.?V 2B2.3, 2B?.n, 2R2.5>. 






Instructional Designer 




; 


The role and responsibilities of an instructional 
designer are neither well-known nor well-defined. Fven 
practitioners may find it hard, to express concisely what they 
do. Therefore let us look at several aspects of instructional 






design in an attempt to define it by interpolation. 




J, 


The ancestors of instructional design (and the sources 


- 




of the jargon and principles used in it) are: 






Programmed Instruction 
Systems Analysis 
• Educational Psychology — especially learning theory 
Educational Technology (audio/visual equipment, etc.) 
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Henc:e., one may expect to find one or more of these 
sklils in- the vita- of an applicant for an instructional 
design position 

There are several things that instruct agonal design 
sounds similar to, but is. distinct from: 

1. Curriculum development may include instructional 
design, but is often more oriented towards the content 
being delivered than towards the strategems or techniques 
' ' for deii'veryz; — ^ 

2i Educational psychology research provides tffe basis 
for much of what: an- instructional designer does, but the 
instr act iprial designer is oriented more toward applica- 
tions than toward formal research . Because CBE employs 
hew and unexplored technologies, however, instructional 
desigriers may well turn up valuable research data. 

Staff wtth broad instructional design skills may be 
hard to find.- Some who have worked only in another medium 
(e.g., pror,rammed texts) may aoply their experience immedi- 
ately and directly, but be slow to adapt or develop techniques 
effective for CBE. 

Aut hor- ' 

In those guidelines we have used the term "author" to 
specify the individual who bears the largest personal responsi 
bility for the development of a lesson. The term "author" 
has been applied to individuals in so many different roles 
and with so vastly different .skills that trying to establish 
a firmer definition of v/hat an "author"' is does not seem to 
be a profitable pastime. The next section describes several 
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typical sources for authors, the advantages and limitations 
of each, arid their availability and/or ease of recruitment 
<3Bl- r>. - 

Many-subject-matter experts-become authors by learning 
the GBE language and beginning, to prepare lesson, material. ^ 
-Because becoming this type of author requires only the qualifi 
nation of subject matter expertise, such staff are compara- 
tively easy to find. However, SMEs suffjer fr^om at least two 
shortcomings. Some of these author recruits find it difficult 
to- learn to program quickly anjd^:eXf iciently . Others, lacking 
experience with teaching ^^dents or with the development of 
fnstructibnal materials, may create lessons which teli rather 
than teach (e.g., their lessons are non-interactive page 
tunning). They may have to rediscover for themselves the 
/principles of effective, efficient instruction <2Cr,1>. 

Some authors are instructors who have been taken from 
the .classroom and taught the GBE language, Eqliipped with 
knowledge about teaching and about the student :Ropul:ati6n 
-they are more valuable and perhaps more expensive than SMEs. 
For example, "robbing" the classroom of itjs experienced staff 
may hot increase financial costs but will exact other cSsts 
in other ways. Former instructors share many of the poten- 
tial problems of the subject matter experts discussed 
above. However, their familiarity with the students is, an 
asset that will allow them to avoid some of the difficulties. 
Authors who retain some traditional instructor duties may 
receive valuable information and insight from their dual role 
<2C1.2, 2C3.1>. . / ^ -'--^ " 

Authors who are primarily programmers can learn ^Ke^^&nE.^^ 
language rapidly and produce their first products well ahead 
of other kinds of author staffs. They may have a different 
under-standing of what a "finished" lesson is, however, since 
revision of programming for. esthetic or educational purposes 
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is not a familiar concept- ThouRh it is--common to teach the 
CBE language to subject matter experts and instructors, it is 
unusual to teach programmers the subject matter or instruc- 
tional .design (except when the skills are rather basic and 
can be quickly taught), thus programmers have this funda- 
mental weakness: they tend to design well-written- programs 
which: fail to teach effectively. Conscientious programmers 
may find they have to spend a large part of their time con- 
sulting with SMEs or instructors, rather Chan programming 
<2C3'.2:>- 

It iS' unusual to use evaluators or instructional 
designers as authors. THey are ordinarily too hard to find 
and .too- expensive for this purpose, 

tbest the reader come to the conclusion that effective 
authors cannot be found, we should, point, out that most of the 
shortcomi.ngs of the various types^of authors listed above can 
be eliminated or alleviated by selecting a site staff with 
balanced abilities and a director who can coordinate the 
sharing of these talents. The purpose of this section is to 
make a project director aware of potential staff weaknesses. 

Other Po si tj^ns 

Programmer, The number of programmers needed depends 
on the degree to which authors are expected io program^their 
own materials. In the case of highly-paid professionals 
(e,g,, M,I),'s, lawyers) or manually-oriented technicians 
(e,R., machinists) it is probably more cost-effective to 
consider employing programmers rather than training the 
SMEs to become proficient programmers. At the beginning of 
any new project, when the staff is new and inexperienced, 
there is often a great ne^d for programming skill. As the 
project or group -^ges, the ratio of programmers to subject 
matter experts may decrease (although the total site 

9o 
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^expertise in prop.ran ,.ing has increased) ♦ When SHEs do none 
.of the first draft programming, a ratio of two or three 
^programmers per SME has bten suggested by several project 
.dlrie^tors <3B1:. 2, 3B1 • 3 , 3B1.i:,^ 302. 2>/ 

'Procto r > Even when a project is too small to justffy 
such a position, the responsibility for this duty should be 
unambiguously assigned^. * For example, rather than stating 
that any staff member present in the^ room is to act as a 
proctor, designate one staff member for specific days of the 
iweek or hours of the day. 

Sub'je cjb ^ m atter expert .re v iewer . Persuading non-CRE > 
.s.taff ;t6 act as reMewers .can :brihg many benefits. Liaison 
is facilitated, rumors, are squelched, lessons are improved, 
and a feeling of working toward and meeting common goaLs is 
promoted • 

Ed itor . Though this may seem a rather small function, 
experience shows that unless the responsibility is assigned 
to someone, small errors will RroatVy d"etrac't from otherwise 
highly-respectable lessons. 

^i?*l?£5l Cpmmen t 

Although this section hopefully provides site managers 
with sone guidance toward selecting a staff, it is very 
probable that one or more staff members will be selected who 
are rather ir ompatible with the job responsibilities. This 
is especially il<ely if the staf^* member s are new to CBE. 
One should ancicipato thio problem and consider how to solve 
or alleviate it. Because new authors* abilities with the CRP* 
language and with instructional design are generally obvious 
during training, It may be prudent to train mor'' ithors than 
-wilj Ultimately be needed. Note that the final lection 
must be 'carefully handled to avoid bad feelings about not 
being chosen. 

9G 
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A selection .prpces^s which may avoid attaching a stigma; 
to people not selected invoices: offering the tnaining to all 
jmembers of an institution. Special invitations and. encourage- 
;meht can be: gtv.eiiJt6_pjeAp.Le_w.hps backgrounds are promising, 
the trainees Who learn quickly and show early potential to be 
;good authors can be offered to be supported by the. QBE project, 
Those performing less well can be allowed to continue without 
support while working tbward their own teaching, goals • Even 
though many ^people are trained and only a few,_uhsu ppprted 
Jtrainees will continue., the CBE training serves to familiar- 
ize the staff of an institution with the techniques and 
bb/jec fives of CBE. 
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. FRACTIONAL APPOINTMEKTS 

This section could be .summarized in one sentence: there 
have: been toa jnrany full time authors and not enough fulirtime 
directors-. Mow for an expansion, of 'thisr terse statement. 

Authors. If current staff members are tb^be selected 
and transferred to a CBE group, it often proves effective 
tp let them keep a fractional commitment to their former 
positron. When course improvement via the introduction of 
CBE is the objective, such a staff member can act as a bridge 
^between current instructors and the CBE group. If an instfuc'- 
tor is involved part time with CBE, he can keep^. abreast of 
dXyelopments^ 1^^^ and: likewise build, stuUent enthu- 

siasm and \find volunteers for lesson, tryouts. Depending qn 
the ipotehtial need for liaison, and^ abilities of an instructor 
toward that end, it may bo advisable to carry his full salary 
but "release" him; to work part time in the course. Contin- 
ued involvement and comrnitment to the course are helpful for 
choosing appropriate topics for CBE, maintaihjjig rapport with 
the rest of ;the instructors and squelching any unpleasant 
rum^ors which arise. Some authors have wished that they had 
duties in addition to those of writing CBE lessons because 
they felt the need. for .other stimulation in order to be. 
creative. 

-Split appoihtmen-ts are not without substantial problems, 
however. Because CBE development is often less structured 
and deadlines' are less firm, staff wtth split appointments 
often find their teaching or other duties encroaching, on 
their CBE time. That is, tests must be written and* lab books 
must be graded. But a CBE lesson revision may be postponed 
or skipped,- and data analysis may be ignored or done in g cur- 
sory manner. Therefore, it is advisable to set aside and 
enforce sep^r'Cte time slots for each kind of endeavor. Sepa- 
rate office locations are often an aid. 

98 



91 

Whea a curriculum is being rapidly dev.eloped by a large 
group,, the time headed for coordination within ^the group is 
typlcaMy so. large that an^ author with less than^ a halftime 
.appoiTitment makes little progress* (This is not. true for 
support staff who need not attend all staff meet ings, ) When 
independent modules are bein& constructed by an instructor 
for his own course, a quarterr or third-time appointment is 
minimal, the topic of what fraction of time is optimal for 
various staff positions is treated at considerable length 
ih <2Cil.1, 2CM.2, 2Cii.3;, 2C5.1, ,2C5.2,, 2C5.3>- 

Director s ; Because good administrative staff are rare 
ahjd expensive^ there is a tendency to spread their talents, 
too thin. iThe director ought to have time to regularly 
see students and staff working. There should be a staff 
member (called director, manager, supervisor,, coordinator,^ 
whatever) who oversees day-to-day- probiems.- Tr there are 
more than about 3 PTEs or 5 people, that person should be^ 
full time,, not half time <2B3. 1:, 2B3.?>v 
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lOUCHSTOHES FOR STAFF SFLFCTIOJr 

General education as evidencedT by academic degrees or other 
awiards seems to be helpful, A firm math, background makes 
learning the QBE language easier. 

Persons who have recently been enr,olled in some sort of 
formal education classes tend .to more quickly learn about 
C8E and how to operate the terminal. 

Master teachers with 20 or mpre years-b^^f experience have 
^been found; to be inyfjluable sources of'*lrif6rtn#t ion about 
students, their problems, crisp examples^ scenarios, etc. 
*bAijt many of them find it diTfilcult xo leara to program* 
Ihstead^ of expec;ting them, to becbme proficient in _the XBE 
language, one .can exploit their knowledge to organize 
content, suggest, lesson strategies and review the final 
products 

A greatly-needed personality characteristic is the abi-Iity 

to take criticism and to learn -thereby — a sort of humility. 

The staff member who portrays a "the cuatomer is, always 

Tight." attitude and encourages constructive criticism is: one 

to choose * The presence of stich an attitude, can be judged 

from the attitudes and practices such as the following.* 

Does the candidate submit only final-typed versions of 
his papers and memois for critiquing by colleagues (and- 
thereby suggest that comments are not really wanted).?^ 
Does; the candidate request critiques at all? Are revis 
ions made on the basis of them? Do student commf^nts on 
test items cause .improvements to 'be made? Are student 
comments taken seriously? As an instructor, has the 
candidate ever sought opinions of others about how to 
teach or test a topic? Does the candidate regularly 
revise classroom handouts? 
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Experience in teaching is naturally: a valuable asset. Note 
that experience in an individualized teachinR format (e.g., " 
tutoring^ TV tapes./programmed instruction) is rriore^. valuable 
than ordinary classroom experience (i'vner, personal communi- 
cation). ^ ' ^ 

Preparing a QBE lesson requires a 'great deal of technical 
writing. A candidate's skills may be assessed by examining 
ibooks and articles. Letters,, tests, arid directions for lab 
exercises, for example, should also be examined because they 
may^ be judged more eas ily: iwithou reT-erence to their subject 
matter merit and are more likely to reflect the ability of 
the candidate- unaided by editors. Of primary importance is 
clarity; next is succinctness. Consider whether the candi- 
date seems to be striving, to impress his_audience or to 
communicate. 

Specifying the level of subject matter expertise is oft 
difficult, t useful rule of thumb (Avner, personal 
communication) is that the^^ lesson^ author sho^uldi be 'Vbne 

^teyel higher" than the students. Thus a person whot just 
completed a Qourse is not. a good; choice for authoring, a 
lesson for that course. A recent student 6an offer very 
valuable insights.^ but not in the role of the. author. 
As a further guide toward -determining what; "one level" 

' means, consider that high school students are considered 
.able to" tutor grade school students, college students able 
to teach hfgh school students. In cases where degrees do not 
apply, an estimate of two to. three years of relevant field? 

^experience (not merely two to three yisars.^sihce training) 
may be equivalent. The topic of students as authors is 
treated in, the next section. 
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:MISCELLAf!E6US CONSIDF.RATIONS " 

Studen ts a s a uth ors . In many academic^ situations, one 
finds a talented pool of relatively inexpensive labor in 'thisi 
students of the institution, For^ a variety of reasons, the 
students -often adapt more quickly to the learning of new 
languages and the use of the CPE system^ than do the faculty. 
In such a situation someone frequently suggests, "Why not 
let the students write the lessons? 

If pro grammin g CBE lessons were identical to writing 
CBE lessons, there would be no^ -problem. However,, unless 
the students have sufficiently more backgroundM^hah .the 
course they are writing for, they are likely to ;h^ve diTfi-. 
culty organizing the material in any way different from that^ 
in whilch they -were taught. Thus juniors and seniors might 
write freshman/sophomore level lessons, but faculty .Should 
write Wessons for the jUnior-s and seniors. Since most stu- 
denies are^ available half time or less, the probl^ems noted 
Previously for part-time work are present. Furthermore, the 
.prev;i.pus suggestions about the advantages of part-time 
employees "assumed , that the rest of the employee's time was^ 
spent in an activity which Gomplemehted the authoring tasks; 
this^ Is unlikely to be true for a student. Finally, exper- 
tl>se built up in students will be lost by graduations 
'unless it can be transferred to other students or permanent 
staff <2ei;3, ^?Ct.4, 3Gi.3, 3C3.1>. 

On the bright side, the low cost of student, labor , :plus 
its avail ability may make^^s'o^ Targe a :pay6ff that the' risks 
are reasonable. It will no doubt be a valuabl^e learning 
experience for the student , and a student may be able to 
introduce and teach faculty about programming more quickly 
and eaisily than otherwise possible.. However, student-written 
lessons ^wiil probably need extensive revisions before they 
are ready for classroom use.. 
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Students' abilities to rapidly acquire proficiency With 
computer languages provide a resourae to be tapped in: two 
ways. First, students can. be nicely used' as, programmer's in 
on-site projects. Second, stpderits can kindle the interest 
of their inHructors' and thereby start new projects. For 
example, several university Sites which require every student, 
programmer to have a faculty sponsor have found that a major- 
ity of their nevf faculty users are people whose^ interest 
wsfs aroused by being a sponsor <3.32.?>. One project which 
paired students with professors to produce lessons was able 
to achieve very low production costs and some remarkably good 
lessons (Grimeis, 1975). 

Matching rank and rolig. It is^ important to match the 
rank, position, and salaries of staff members to their roles — 
especially for support staff. For example, when instructional 
designef^ or programmers are held in iow esteem by virtue of 
their lack of formai *'"stahding,'" their inputs (no matter how 
valuable) tend to. be brushed aside.. This situation can arise 
easily when the subject matter experts are expensive .profess-r 
ionals such as lawyers, M.D.s, etc. Ne;ither^fun-pr6fess6rs 
nor four-star generals have the breadth of knowledge to 
ignore comments from support staff. Unfortunately, they may 
have authority to do so <2C2.M>. 

If possible, the most experienced and qualifieci: instruc- 
tional designers or evaluators should, have a higher rank than> 
the authors. In several .projects where this hierarchy was 
reverses, the effect was detrimental. 
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SUGGESTIONS FOR AN AUTHOR TRAINING PROGRAM 

'Give the new autHor a student signon, to a list of less.ons 
whicH must be completed, with a passing score. Make sure the 
list includes both good, and ooor .examples of CBE, This- is . 
a first step for promoting the author's empathy for the 
student. 

If possible, pair up a new trainee with an experienced 
and successful author. If no ^*oid hands" are available^^ 
pair up two new authors and /or a^-non-CBE colieague^ Set up a 
schejdule so that the group meets two to three times per week 

';tp go over progress, problems, and ideas. Verify that meet- 
ings take place for the first several mpnths. the important 
po.int here is that the author grows accustomed, to regularly 
showing and sharing his plans and' products with others-, this 
series of .planned meetings will hopefully avoid the "hermi^t 

^uithtrp^^' ^phenomenon:. , 

Advdse new authors- to expect that they will have to try Jtheir 
lesson materials oh students at early and^ regular intervals, 
the first trial should come no later than eight weeks after 
the^ -beginning: of training. Authors cannot restrict their 
learning to- formal sources such as manuals, Consultants, and 
research reports. They must also learn from contact wrth 
st^udents. Encourage or force authors to test incomplete seg- 
ments' of lessons. 

Insist that the first lesson be ^'finished" or set aside and^ 
the- second lesson started after two to three mpnths . lEncour- 
age authors to prepare written -(especially on-line) notes and 
obmments as part of their preparation for reviewing other 
authprs* lessons. 
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Hav.e the instructional dFeslRher work with every ne'w' author at 
least once; a week. When student testing bep/in?;! the evalua- 
tor :sh6ul4 join the. meeting, (to suggest data collection and 
iaterpretation techniques) 

Provide access ^to on-line typing lessons for authors Who 
are not touch typists. (These lessons are not ordinarily 
needed, by students, however.) 

Provide the new author with a repertoire of known, tested 
strategies and instructional design approaches. Such a 
repertoire can allow a '^novice to develop high , quality lessons^ 
rather .quickly. ^ 

If tr alining can be given at a major CBE center, make sure 
that: authors visit operating classrooms; staff meet poten- 
tial on-line colleagues, reviewers^ etc staff meet various 
syst^^bnsult.ants , maintenance staff, business office staff, 
eyaluators, etc.; authors, instruotipnal designers, and 
evaluators visit the CBE cen ter^ s* library to find books and 
journal suggestions for their home library. 

Concentrate the iritfoduction to CBE and language training in. 
as short amount of time as possible. Although it is possible 
to teach ^bout CBE at a rate of two to three hours per week. 
It is far more efficient to schedule a concentrated course 
^cljjring a vacation period or even a marathon weekend (with 
follow-up training at a slower paCe):. 

It is preferable that project objectives, content decisions, 
responsibilities, course .revision plans^ etc be settled 
completely before training starts. The period after initial 
training i-s a gpod time to review such decisions,, but too 
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much tira^ spent then diills enthusiasm and allows forgetting 
of recentiy^learned programming skills. 

if ^staff experienced in CBE can be employed as consultants, 
the best time to have them is during planning period. If 
they are familiai; with the specific GBE system tp be implemen 
ted, they should be employed during the jfirst weeks following 
training. 

If som'e -Staff arrive early or have free time prior to train- 
ng (arid access. to CBE terminals), they should be directed to 
prepared written reviews of existing materials in their con- 
tent field. These should not be done from a "browsing" or 
"can-T-use-it" perspective, ,but rather as critiques of good 
and bad s.tyles, things to emulate vand things to avoid . 

If the staff is^ composed of members who retain their home 
deparj^mental affiliations, efforts to foster a group spirit 
are useful. Seminars with 'guest speakers, luncheons etc., 
might be considered. 

If high, qua3ity lessons written, at another site would be 
useful for students iri the target courseCs) , try to begin 
to ijse them as- soon as possible. By using weld-tested 
lessons from another site,, new authors will get a^ feeling 
Tor how a class should react to their Tessons after the.y are 
polished. 

Note: Additional speci'fic recommendations for author training 
on the PLATO IV system are found in Francis ,(1976). 
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PART I H-RECOMf€NDATIONS FOR MANAGING A CBE SITE 
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MANAGING FORMATIVE DEVELOPMENT (PROpUCTlbN. STEPS) 

The foiidwing are commonly-observed problems at CBE 
deyerqpment" sites* 

Co lleague review s. As notecJ^ by .Francis, Goldstein, i 
Call-Himwich(1975) and Call-Himwich (1976), authors have 
tended' to wait too long before showing their lessons to 
colleagues for a formal or informaT review* In some cases 
when reviews did take place, problems and errors, noted during 
the. review were nevertheless left unrepaired. Both . the^-.nef- 
erences cited above contain detailed suggestions for encour- 
aging positive, constructive on-site or off-site reviews of 
lessons aB1.:r, MB1:,2, ^IBI . 3, 481 .4>. * 

Student testing . Left to their own instincts, inexperi- 
enced^ authors also tend to wait far too long before testing 
lessons or lessoVi: segments on students. They always have 
••just one more display" to finish before^getting student ' 
feedback.^. Firm managerial control can avoid the problems 
a'ssp^jiated with too much author effort Without stu^ient 
feedback <4B2. 1, 4B2.2, 4B2.4, ^182. 5, 4B2.6>. 

Student respons e . d ata . Most authors collect sufficient 
high-quality data about their lessons to provide a very 
valuable basis for lesson revision. However, because^ some of 
the techniques for analysts: are new to them ^nd because the 
storage space available seldom allows a leisurely investiga-? 
tion of the data^, some authors adopt, a "print-and-stack" 
philosophy. In order to preserve the data,, they get printouts 
of it, then delete the on-line copy. Although this procedure 
is :preferable' to simply deleting the data, it often lulls the 
author into feeling his 36b, is done. He may intend to 
examine the data someday., but in fact, that rarely- happens. 
The hardcopy version is not comp.uter-sear chable , and' the 
author ^s memory and enthusiasm for correcting problems 
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diminishes with time. Soon all he has is a stack of paper. 
The manager of a. site must hot only allow time 'for examina- 
tion of data but also must see that a site staff ^member 
'Experienced in: using student response data guides other 
authors T.h the use and, ihterpretatibh of their data (Francis 
Weaver, 1977 ) <nn2. 3> / , 

R evision . The above three topics: deal with various 
aspects of obtaining f. Redback about lessons to enable the 
author to make an effective and' efficient revision of the 
I^essorii Muchi of the effectiveness of CBE is possible 
becausie of the rapid and specific feedback available from 
s.tuden.t tryouts. For strategies: and topics where the medium 
^ offers : rio extraordinary advantage, one must expect tHat, the 
first version, of a lesson will be no better tarid possibly 
pporery than the .same material in a convent ional medium. 
However, by. obtaining and^ using the feedback available^ an ' 
author can. continually improve the CBE lesson until it is 
^superior to a similar lesson written in a medium without 
.such thorough, sonsitive feedback ( Call-Himwich A Steinberg, 
1977)., 

In most other mediae revisions are difficult to. make 
Cespecially after ".publication"), data is mqre expensive to 
collect and analyze, and: revised forms appear at intervals 
of four to six yearns. With CBE,. each new class offers the 
potential for testing ^a new revised' version with relatively 
iiittle effort,. 

It may be helpful to suggest to authors that they vievf 
a lesson- as an hypothesis about how to teach the material. An 
hypotheses may^ -be found false or in need of modification 
without criticism of its formulator. Considering a lesson 
.as an hypothesis de-emphasizes the "grind 'em out," one-shot ' 
production mode which some authors too readily adopt. 

1^ 
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Lesson integr at ion ;, the formalities of listing and: 
testings prerequisites, standardizing terminology, cross- 
referenc^.ng, and coordinating lesson content are often 
perceiv>ed to^ be rather bothersome and unimportant to an 
author "being creati^^e" with CBE. Howe,ver, unless these 
topi^cs^are attended to, the. result .Will be difficult to mold' 
into a curriculum and will remain just a collection of- topics 
pn :a^ subject. 

Standards , The need for lesson standards and lesson 
documentation has been discussed in part I. Unfortunately, 
such standards are frequently outlined, discussed, and 
adopted: at the beginning, of a project but ignored ey.er after^ 

Produc ti on goals . All the previous items might be 
solvable if authors cpuid be given additional time to prepare 
each lesson. ;Although realistic^ timelines, are essential, 
both over-generous time allocations and the lack of short- 
term goals are a disservice to developers and a .waste, of 
resources . 

CBE suffers from a lack o^ reliable production yard- 
sticks: any two ''topics," "teaching points," or "objectives" 
can differ widely in the .effort needed to convert them into 
an effective,, efficient CBE lesson. "Student contact hours 
produced" is no better a measure because it rewards the 
production of long-winded, inefficient materials. Neverthe- 
less, a manager must set goals and deadlines , -for authors, 
estimating the difficulty of the tasks and the abilities, of 
^the author. It is difficult, but necessary, to strike a 
balance between a too-rigid enforcement of deadline.s which 
frustrates authors- and lowers quality and a too-relaxed 
observance of undefined goals which may allow some authors to 
make a career of a single lesson. 

Liaison . Liaison problems often drive authors into 
seclusion: it is easy to imagine a committee haggling over 
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the content of a. lesson, course, etc. Normally an instruc- 
tor simply teaches as he wishes even though fellow instructors 
may teach somewhat differently. This "leave me alQnev so I 
can do it my way'? approach wonks reasonably well in a lecture 
format where the instructors are competent and are respohs^- 
bre for all the teaching. However, such an attitude in a 
multiple-author project means the lessons may not be^ coordi- 
nated; in a one-person .project, it means that few other 
instructors Will ever take advantage of .the large courseware 
development effort made by the author. 
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CLASSROOM MANAGEMENT 
to what extent will recreational uses be permitted? 



Site directors, usually answer this question: "To the 
extent that recreational uses don* t interfere with other 
uses;" The following ^suggestions: are an attempt to put this 
general answer into a specific, practical format so that 
serious workers taking a break will ;not; feei :gullty, but 
nphrproductive. users will not compete with or detract from 
site, pperatrons and 'goals. 

term in als * Tf there are recreat i'ohal users present, 
there should always be at least one additional terminal free 
sp that a potential user do:esr"not have to dislodge a recrea- 
tional user and/or wait to ?have the user finish. Because 
game players may switch to nonvrecreatipnal uses when asked 
ta leave, and because priority users may- leave rather than 
confront a recreational user.j the "waste" of a free termina! 
must be tolerated by secondrpribr ity users. If a site is. 
large (more than 12 terminals)' it has oiften been? found 
appropriate to^ have two or .more vacant terminals whenever 
repreationai users are present. A site policy^ regarding 
terminal uise* can be partially implemented pn the PLATO system 
via. the -I'current usage controller" or CUC. 

cEC S usage . The use of lesson, stprage space (ECS) can 
also be controlled by the CUC of the PLAt6 system. Users 
working in their own lesson spaces* can^ however, consume 
space that would be used by high priority users, were it 
available. Therefore, the site proctor of a general-service 
sHe must regularly check the contents of unfamiliar lessons. 
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Hoj. se and. clutte r , it Has been, widely observed that 
recreational users tend to rhave a variety of habits which 
may be bothersofne to other users.. For example,, many games, 
require or encourage rapid banging on keysets. Because 
recTeational users are frequently limited to laternight and 
ear4y>m6rhing hours, food- wrappers* bottles, arid other debris, 
may accumulate hear the terminars!. 

Opini on- and Im pr ession .. On.e of the greatest detriments 
caused by excessive recreational use is that people who are 
casually aAquainted with: a site may begin to associate CBE 
with recreational use,, i.e/, non-academic, noa-;serious 
endeavor. Fortunately;, this negative ImpressipA is usually 
formed only when game playing is excessive (i.e., to the 
practical exclusiofirof more legitimate uses).; the occasional 
use of recreational programs as a diversion by hard-working 
staf f seldom leaves a bad: Impression . 



Who can give demonstrations? When may they be 
scheduled?- Can site resources (such as terminals) be 
reserved or given priority during demonstrations? 



Because of the novelty of QBE, there is a tendency by 
site staff and higher administrators to '?show It off" rather 
frequently. Unless this tendency is controlled and managed, 
a large amount of resources can he spent without furthering 
the objectives of the project or site. At one site in. our 
experience, the entire staff lost a day's work preparing, 
waiting for, and re-hashing the outcome of each demonstration. 
A separate demonstration room with, a limited-length standard 
demonstration is a partial solution to this problem. 
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MAINTAINING STAFF MORALE- 

The foi lowing is a list of suggestions for rewarding 
individual staff members. It. is often desirable to grant 
small, non-monetary rewards for hard work, competence, or 
other exemplary habits. Like students, staff members respond- 
to, postti^ve feedback, the list below includes tasks which 
-provide strong interna! rewards arid thus cbntri.bute to 
overall job satisfaction and also management actions iwhich 
can provide reinf drain g feedback to employees. 

if a lesson fs working reasonably welL,^ the author :generally: 
gets, an ego boost each time students use it. The student 
and instructor comments will always include a few new compli- 
ments arid the (hopefully) rather minor corrections needed 
will remind the author of Just how ^polished: the lesson has 
become. 

The respect of coTleagues is- sought after by staff in all 
fields:. The PLATO sys.teni offers an additional dimension : 
on-line^ colleagues. Because of the PLATO system's communica- 
tion features, it is possible to become acquairited without 
ever meeting f ace-to-face . This allows a relatively inexpen- 
sive method for exchanging iprof essional opinions and. finding 
additional users .for courseware! Since it is a compliment to 
have one's courseware- in use in wide circles, 'the PLATO sys- 
tem's geographical -dispersion .provides a "coast-to-coast" 
vehicle for rewarding the author; 

Recognition of the contributions of staff cari be made ..on the 
title page of a lesson; in site brochures, re^ports,. and pro- 
posals; and in references and corauthor ships for journal 
articles=. 
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For most staffs it is a compliment to be asked: to give- a 
demonstration *pf the ^CRE system and the courseware of the 
site. Gi;Ving demonstrations (or speeches) or being a project 
representative at meetings carries a reward in proportion« to 
the responsibility the task requires. A smaller but still 
substantral reinforcement is given when a staff member *s 
lesson Is uked. by someone else for demonstrations. 

.Lesson^ reviews, because of their basic nature tend to be disr 
couragihg to receive. The most important way of redu^ijig,^__ 
the negative aspects of lesson reviews is to avoid giving 
an^ author a review which suggests massr revisions. This 
can be dope by "providing, frequent, careful monitoring of the 
content and instructional design.. Several short reviews 
throughout the development of the lesson are more useful and 
more palatable than a massive review at the end. 

^'^Releasing scarce resources such, as signons and lesson s^ace 
for personal use by staff members (or their families) is; a 
.substantial reward. - 

iMore than in most environments, the work that one does oh a 
CBE- system^ can be performed at any Jour of the day. If qot 
ordination, requiremenj^s are low, it may be possible to offer 
extremely flexible working schedules. In fact,, because the 
potential flexibility l^^s obvious to all, failure to offer 
ft may produce discontent. 



115 



108 



EVALUAfipN ISSUES 

, The .two obvious direcfions for an evaluation are a 
coijiparison of ^the- CBE materials against a pre-established . 
goal or their comparison to ah existing situation., 

Nbn^companati ve Evaluations 

The well-known difficulties of insuring exactly equiv.a- 
lent conditions for a-"hew vs. old" comparison leads some 
people to make a "new vs. goal" evaluation. That is, rather 
than comparing the new CBE materials to\the t^ non- 
CBE course, they measure the extent to which the CBE materials 
met the objec.tives stated . Some ^people feel ?that they can 
thus avoids setting up a> cpmplex research destgn , dispense 
with the interpretation: 6f statistical, analyses , and generally 
eliminate the hypotheses.,, assumptions, control groups y and 
detailed -reports 'associated with a "-formal comparative 
analysis." they replace these diffVicult and^ time-consuming 
procedures with production or operating goals and then merely 
determine "if they made it." 

For example,, under this mode of operation, objectives 
might be to "produce 10 role-cqnflict simulations" or "teach 
too students to use the Wheeler-^WhTtson apparatus each year 
via^ CBE." The evaluation of the latter projects seems simpler 
one determines if the simulations •have been, written or if the 
students have completed the lessons. 

Although this form of evaluation may be appropriate 
...during the early stages of an evaluation^, its use as a sole^ 
summative measure is shortsighted; One simply cannot avoid 
a comparative evaluation. Comparisons may be made with 
little ^basis and few facts,, but students, colleagues, and 
administrators wi~n decider Was it worth the "effort?. 
Should we continue? , etc. Skeptics may suggest alternative 
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uses- Tor the monetary and- personnel resources, use of a ^ 
•different CBE* system or a totally non^CBE medium, adoption of 
a different curricular or instructional ^desi^n philosophy,, , 
or implementation under different conditions. For example, a 
site may "successfully" create its 10 sirnulations, only to 
find them challenged as unrealistic, :nqtrrequiring/CBE, or 
borinc. Or the simul;ations may be highly respected,, but too 
expensive to support. 

Because; administrators in some fields -(c.g^.^ engineering) 
may be used to accepting as "success" the fact that something 
.new has been— found pr created, they may aiapw "meets the 
^objectiyes" evaluations initially. However, comparisons with 
^alternatives are ultimately ineyitabre.. Therefore , because 
'of this inevitability, project staff should be encouraged to 
gather information and prepare an evaluation so that the 
predictable evaluation questions can be answered^ on the ba^is 
^of Well-documented facts rather than on specul^atipn • 

* Comparati ve Ev alugtioTis 

The watchword for making comparatfve evaluations- is 
(what else!) "objectives." Unless: you carefully consider 
the objectives of the evaluation^, you may be saddled^ with 
one of several common problems. 

the first problem is trying to measure (and compare) 
' .everythin g. As noted in the discussion in^ part I„ there are 
a multitude of comparisons that can. be made. Only a few are 
relevant and importarTt in^ each, situation. Rather than 
spreading an evaluation effort too thin in order to measure-, 
many parameters, one should examine the objectives of the 
evaluation to determine which measurements, are critical , 
and concentrate on those. A. set of contingency plans for 
the, evaluation should, be .drawn up simultaneously. 
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' ^he rsecqnd .problem is often found t'p. 'be iT^ 
:aften it is investigated;, but it neyer.theiess produces great 
-frustration until that time. Here is the. apparent dilemma: 
If 'YOu;'use CBE to teach topics in the way. you used^to, you 
haven't exploited tts potential and it will probably not be 
cost-effective for most applications. However, in this 
situation the testing is comparatiyely simple: you. can use 
the same tests that were used before for CRE-taught students. 
On the other hand^ if "you "ule CBE lessons which teach at a, 
higher cognitive level via simulations^ mr/deling, role- 
playing and other techniques which- require a computer, yon 
have two choices with respect to testing.. If you use the old 
prerCBE tests, the> will be a fair comparison for the 
original topics, but will not indicate that students taught 
via CBE may have, learned new, high-level skills*. If you 
write a new test Which measures the high-level or CBE-taught 
skills, it will not be possible, to compare results from the 
tyrp; different tests. It is not fair to give the new test to 
students taught without the CBE lesSohs because: the CRE- 
taught students would have an unfair advantage. 

The way out of the above dilemma is (naturally)' to 
re-examine the project objectives to answer the following 
question: "Are: the old objectives satisfactory as- ^the-y- stand 
(and are you thus looking for an improvement in teaching 
t€|Chniques, motivation ^ retention, etc.) or are^ the objectives 
inadequate or obfjolete? If the. old tests are good tests of 
the objectives and are reliable, they should^ be kept-. If the 
objectives have changed or if the old exams do not adequately 
measure* them, test development is needed .. 

E valuatin g "CBE " 

It should be not.ed that it is very difficult to "test 
the effectiveness of CBE itself" apart from the lessons, 



strategies, and tmpi^mentation of the CBE system. This 
.is: .ahothef way of saying that CBE is a, medium, not a treat- 
ment. One can evaluate the ability of a CBE .system to allow 
v.arious teaching strategies and: kinds of lessons, and to .. 
facilitate particular implementations,, but most other evalu- 
ation questions depend on the way in which the lessons were 
.written or implemented. Thus the question, "Did the CBE 
system increase, the student scores?" should- be rephrased^ 
"Did the lessons written for the CBE system- increase student 
scores?" As reformulated-, the question suggests the possi- 
biMty that another set of lessons mrght have done a better 
or poorer job of increasing student scores. It thus clarifies 
the focus of the evaluation and hopefully prevents post hoc 
speculation about whether the results Were obtained because 
6;f or in sptte. of the CBE implementation. 
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THE PROJECT MANAGER.^S fLIST OF WARNING SIGNALS 

The director of a CBE project and the supervisor of, such 
a director should both be alert for "smoke" that indicates 
smoid.ering "fires" within a CBE development effort. The list 
assembled below is not complete, .but is useful bo.th^as it 
stands; and becaiisie- it .suggests other danger signals. 

If touch input, microfiche,, and audio output are incorporated 
into^ lessons, but are not used twhen testing the student, their 
^use" (hay be trivial and decorative , only. If these features 
;are^ needed to teach, they are normally also heeded for testr 
in|. Similarly, lessons which use animations, simulations, 
and case studies to teach probably need these techniques to 
test. If they do n6t| a 'brief investigation may be^ . ^ , 
warranted. \ ^ ' 

i If microfiche, images and, drawings are presented with no 
questions to check the students understanding of them (e.g., 
"Here is a framis") the high cost of producing these illus- 
trations is not being recaptured by making effective u'se 
of them. Likewise, check to see if expensive drawings, 
attachmeri -^s , ahd recordings are being used more than once. 
Can a drawing that costs $250 to create and reproduce be 
justified if students view it for only a :f ew seconds? 

If you can complete a lesson by successively trying each 
multiple choice answer ^or by entering nonsense ansWers^ until 
you get the correct answer, the quality and validity of the 
lesson should be examined. Clearly, a student *s completion, 
of such a lesson, indicates little unless the lesson is 
followed by a comprehensive test. For most students, lessons 
wi-th "paths of least resistance" such as those outlined above 
produce robot-like behavior but little learning. 
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If Student response and area summary data are not examined • 
wit-.hin a week or two after they are collected, and are merely 
^stored on-line or printed^ out and stacked for later reyiBipn, 
there are serious questions about the, .ef fectiveness and' 
efficiency of lesson revision. It is a simple matter to ask 
an author a few questions shortly after students' have used a 
lesson he' wrote^. It is appropriate to ask to see where the 
data is, what analyses h9ve been made, and what changes hay.e 
been incorporated: into the lesson. Other useful inquiries 
are, "How long does it jtake to look through that data on- 
line?**, "Dp you analyze each stud;ent run?*?, and "What changes 
have you made on the: basis of .student response data?^ 

iP'student notes or comments file is normaliy attached to a 
lesson, or to a courise in order to gather student suggestions 
about a lesson. Reviewing these comments can be insightful., 
though all comments cannot: be taken; at face value. If some 
of the suggestions have^ basTs or merit, a discussion of the 
student views with the author may^ indicate the author's' 
view toward students, The presence of authors who ignore 
s t«Jd en t comment out of hand indicates significiant problems. 

By asking, an author about the topic .and status of' lessons 
written by his cpXleagues , :you may be able to simuataneousTy 
make two observations. Pirst, the author's answer Indicateis 
the level of involvement he has witb others, e.g., if the 
author has become a hermijt. Second, it indicates the level 
of inter-colleague reviewing. Note that the question should 
not force the author to render ah opinion about the qualit y 
of the rest of the staff, just .w hat they are doing. Also, it 
may be useful to ask the author if he has reviewed any 
partially.-c6mpleted. lessons. Aythprs who :are requesting 
reviews during (rather than after) development should be 
reinforced for this action. ' 
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If several rnontThs after tra-tn-ihg, aa author cannot explain 
why he is writing objectives or documehting the lesson, fur- 
ther training and guidancje are needed; Likewise, if ah author 
cannot show the orderly relationship between thef objectives,, 
the test, and the body of his lesson, the manager must reluc- 
tantly conclude that the instructional design training hasn't, 
been understood or accept ed\ 

The nu."nber of students who can be handled by a single proctor 
is a good indicatlun of the degree of polish of the lessons. 
Though "polished" lessons are not necessarily good lessons^ 
any lessons which haye been used by more than^ 50 students and 
which still require heavy proctoring (fewer than 12 students 
perjjroctor) are probably not getting the attention they 
n eed; 

There is a set of comm6nly-use0 computer routines on the 
PLATO: system in what^ is- called the ^'library." If authors of 3 
tp 6 months of experience don*t know of the existence or the 
Qontehtis of the '^library^" they may be spending time re- 
inventing routines that are already available; A review of 
on-line resources is appropriate* 

A list of ail the lessons being; developed or in use i§. found, 
on the pLaTO system in the "catalog" lessons To be sure 
that authors are not ignoring already-available courseware, 
a manager should ask* the author about o^ther materials avail- 
able in the subject area , the relationship of those materials 
to lessons currently being developed by the author, and the 
feasibility of incorporating segments of already-written 
lessons into new; materials. 
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MISINTERPRETATIONS 

Two misinterpretations have been observed often enough 
to warrant warning site managers about them. 

Some authors misinterpreted the zea! with, which CERL 
instructional design consultants promoted data^ collection,^ 
use p£ answer-judging, rath.er than multiple choice, and inclu- 
sion, of feedback for speci_fic wrong -answers. They felt, these 
things were being promoted by \CERL staff because CERL wanted 
^powerful features of the system ejcpipited for organizational: 
glory. In: fact the CERL staff fei:t that these were funda- 
mental f eatunea (not esoteric ones) and. that if .such> features 
were available, but unused, there was little justification or 
^need for employing a medium .as expensive as CBE., Only ^ 
through use of the powerful software features can one justify 
using CBE. 

Authors misinterpreted reviewers' critiques of the 
positioning, of text, arrows,, and feedback- as well as their 
implementation of touch panel input. Authors tended to 
regard these issues as unimportant esthetic considerations. 
Xhey felt that so^ long as the subject matter was present and 
accurate, there was little need for further revision. In 
fact, CERL and other groups have accumulated substantial 
evidence to indicate that one can efficiently "hide" infor- 
matibn on the screen by placing it incorrectly and that 
casual implementation of features- such as touch can signifi- 
cantly reduce test scores (Avner, personal communication; 
Venezky, R., Bernard, L., Chicone, S. , & Leslie, R.., 1975). 
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POtFJlTJALLT CpST-RFFECf USES OF CBE AND PLATO 

A CBE. system itself is neither cost-effective nor non- 
cost-effect i:ye. It is ohcly/ the uses to which- the system is 
put Jthat determine, cost-effectiveness. To guide new sites 
and pro jects towards^ uses of the CBE. ^system which are more 
likely than average to be cost-effective,, we have prepared 
the suggestions below. 

Simulations of expensive training equipment (including hard- 
ware simulators) Jia^ found to be co st- ef f ec ti ve 
terawford, Hurldck, Padilla fc Sassano, 1976). 

It is a mistake to discard all conventional teaching materials 
and replace them/ all with CBE lessons. Often workbooks, film- 
strips, video tapes:, or programmed texts can be supplemented 
with CBEi By supplementing, high development costs are 
avoided, the high cost of equipment and materials continues 

to be amortized, and the rate of preparation and validation 

of the improved instruction is increased;. 'For example., if a 
video-taped training movie has become incomplete and obsolete 
after three years (or if students don't seem to completely 
comprehend it), the tape could be followed with a short CBE 
lesson which covers a bit of new materi-al and corrects some 
obsolete nomenclature.. Then an interactive quiz over both 
the tape and the QBE supplement could be administered. Even 
if some comparatively expensive CBE animations must be pro- 
grammed, that may be cheaper than re-shooting the entire 
video tape.. Because many "home-made" training movies forget * 
to point out-a mechanism , ''note an exception, or focus 
attention at the right time and. place, they sometimes, fall 
short of their potential. A small in vestment in supplemental 
CBE training might thus yield large instructional benefits. 
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Because of the PLATO system's high flexibility, it may be 
advantageous to design? other computer teaching systems or 
instructional materials on PLATO, even if the final product 
is to be delivered by a less sophisticated computer or medi— 
.um.. Tor example, the> evaluation and validation of instruc- 
tional materials, (including tests) may require several itera- 
tionB .of product development ,^ the gathering and analysis 
of substantial data, and ithe preparation pf detailed docu- 
mentation and reports. It may be cbstref.fective to do all 
the development w>»rk on a PLATO system, then: prrnt the mater- 
ials in final, validated form .as conventional or :programmed 
texts. Similarly, a GMI system capable of being implemented 
on a mini-computer might be ;better designed on a powerful 
computer such as the PLATO system, thsn converted for the 
smaller, cheaper computer. By using the PLATO- system initial- 
ly, some options which .would be difficult to install on the 
mini-cpmputer could, be, amplemerited: only those options which 
were found to have great value wouid ultimately be installed 
oh the mini-computer ; 

V/hen group pacing is being used with conventional materials 
and media, it may be cost effective to provide individualized 
remedial training for slow members of the group during evening 
or free hours. Since in group .pacing the speed of the group^ 
Is set by the slowest member, a relatively high cost for a 
few students can be offset by the; savings of several hours by 
all members^ of the class:. 

Similarly, when costs for drop-puts are high, remedial courses 
via CBE may be cost-effective. In some circumstances it may 
be possible to combine two courses teaching, the same topic 
to students with different backgrounds (e.g., with and without 
a strong math background) by providing an individualized 
supplement for some of the students. 
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There are two approaches for analyzi'iig the question: "What 
size -course is optimal for G"3E lesson implementation?" By 
focusing on development costs, one can show that a course 
Which has many .students can more quickly amortize these 
costs. By focusing on deliysry costs , one can demonstrate 
that instructor costs fqr small classes are very high .and an 
easy target for cost effective CBE lessons. Furthermore, 
small ^classes are typically :taught infrequently , . thereby 
inconveniericing students andr^perhaps lowering total enrollment. 
Once developed, a low ^enrollment , self-instructional class 
might be quite '.inexpensive to deliver and may provide high 
.tpy.al.tles dr pther small institutions wish to 6ffer the 
course -to their students. If cost ef fec.tiveness is -to be 
achieved via time savings, one should choose no.t only a 
course where substantial time can be saved, but also a course 
whose students are being: wel^lVpaid for their time. Jor exam- 
ple, the money earned hy savAng one hour of an executive's 
.time probably pays for more hours of CBE time than does 
saving one hour pf a clerk's time. 

In/ the situation of the impending loss of an irreplaceable 
instructor, CBE may be able to catch his best mqmentis, most 
thQught-provoking problems, and most efficient teaching itrat- 
egies-^and do so far more effectively than any other medium. 

Travel expenses for continuing education, updates on new 
technology,; etc. can be reduced or eliminated by implementing 
on-site continuing education (or refresher) courses via CBE* 
In some professions, however, trips to take refresher courses 
are treated more as vacations than as training sessions.: In 
such< cases, a CBE replacement for these trips will hot be 
'popular. 
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Some testing situations require the. simultaneoujsu testing of 
8; team of students (e;g. , replace a truck motor) Because 
above average students may compensate for weaker students,, 
individual competencies and weaknesses are difficuit to find., 
reward,' or corrects Such performance tests are typically 
Ipng and expensive to administer arid difficuit to grade; 
CBE simulations offer a way to test for each student the 
cognitive pprtioris of the task (e.g^. , order of procedures) 
with less cuing and ^prompting present than with other media. 

Training in, safety and safer-practices is often done perfunc- 
torily, with no enthusiasm, and llttTe effect in many indus- 
tries. ^Unions: and government bodies require monthly safety 
meetings which are sometimes more of a "rest period" than a 
learning experience. CBE. lessons offer the. possibility of 
simulations for procedural tasks which include "buil^t-in, " 
random calamities to be dealt with. Accident prevention can 
result if the training can be shown to have a measurable 
impact on daily procedures and. personal behavior. 

Calculations about costr-ef fectiveness must include all the 
constraints and conditions. A good, example of how an overe 
simplified comparison might produce misleading results is 
found in Lackman (1975). In Lackman's study, medical simula- 
tions were prepared in both PLATO CBE format arid as paper- 
and-pencil exercises. When one calculates the development 
costs for the simulations in each medium, one firids the CBE 
format costs 4.7 times morer and thus appears to be the most 
expensive. However, the simulations were used by practicing 
ll.D. 's at conventions, professional .meetings., and informal 
courses--aiways with voluntary participation. The actual use 
of the PLATO CBE simulations was found to. be 9 times greater 
than that for? paper-and-pencil simulations. Hence, on a "per 
use" basis, the development costs were only half as much. In 
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a similar fashion, one must carefully consider !iow best to 
calculate cost/hour, for CBE materials.- If phe spends^ time to 
'Improve a lesson by allowing students to complete the lesson 
more quickly, the change makes the,.,.QOst/hour less favorabl.e 
.even though the lesson is better. 

When, considering the cost effectiveness of CBE. compared to 
other media or alternatives, consider what options .are actu- 
ally available.. Fo,r example, it may be cheaper to use another 
computer to do some numerical integrations, :but if the other 
con^jiuter is blocks away or offers only overnight service:, a 
CBE system with/higher costs;, but fas-ter turnaround might be 
preferable. Likewise, video tape might be the most appropri-. 
ate medium for a- topic, but if you lack video equipment, CBE 
animations might be the most cost-effective option available. 

It may be more cost-erfective to^ write hew courses using CBE 
than to convert existing courses. This is particularly true 
when, substantial capital costs for equipment,^ texts, rooms, 
etc. have already been .made. Even if a CBE-simulated lab 
Is cheaper and teaches better than actual lab experience-, the- 
fact 'that the cost of existing lab equipment and special, class 
rooms has not yet been amortized may mean that switching to 
CBE simulations is impractical. 

Much of the time savings arrived at by converting a cbnven- 
-t^ionai lock-step, lecture-format class to one which is comr 
puter-based are due to time saved through self-pacing. Some 
opponents of CBE argue that only the savings of CBE over and 
above Jthat for other self-paced media can be considered when 
calculating cost-effectiveness. However, the management 
func^tion of CBE is often the key to being able to implement 
-.and^ .operate .a self-paced system, that is, without CBE's 
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mahageme^nt; capabilit^^^ would hot be feasible in some 
cases to design a. self-paced course using other cheaper 
media. 
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WHY USE CBE SYSTEM LIKE PLATO? 

The decision to use CBE, and the PLATO system in partic- 
Ular, may' already have been made when the site manager assumes 
^his-duties. Nevertheless, it may be prudent for the manager 
to review the decision. Below is a brief list of\character- 
istics of the PLAtO system which make it worth using. 

It is a computer and can dp things computers dp. 

For example, it can :store and process data.. On 
the^ other hand, it: is not a general-purpose com- 
^ " puter: it has restricted database operations and 
relatively slow input and. output f rom termiriars'. 

It has a uniquely powerful iarig;Uage f;6r teaching and 

performing other special functipns. 

TUTOR is^ widely recognized and. imitated because 
of its ppwerful features. Moreover, it continues 
to ad d^ new;, useful features . Hpwever , TUTOR 
programs are not readily, transferable to other 
computers. 

The PLATO system is equipped with a unique graphics 

terminal as an output device. 

The terminal and the auxiliary devices which can 
be attached to it pffer a broad potential for 
nearly any important kind of interaction with ab- 
user. However, unless the training requires 
these features and the lesson developers make 
. use of them, the potential i5_ wasted. 

The communications network providecf by the PLATO; system' 

is unmatched in terms of the breadth of types of users 

and types of information communicated. 

Lesson writers can not only share courseware, but 
can co-develop lessons, review lessons during 
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d.eveippment , etc. Nevertheless, there continues 
to be duplication of effort .and missed opportuni- 
ties for inter-site communication., 
* . . - _ _ ^ 

-The ability of the PLATO terminal to control, external 

devices ^makes it ideal for running psychology/educatiori^ 
experiments . 

Student interaction data related to educational. 

goals can be gathered rather automatically ^ 

Timing,_presentatioh of grapM^ and other stimuli 

can. be int^i^spti vely conthoiled. 
Both the hardware and software are designed so that a 
:student can begin to use^the terminal after 2-3 minutes 
of orientation. No computer jargon^ , programming- skills 
pr technical abilities are required. 

One can-find somewhere on another computer or in some 
other medium most of the features cited above (with some 
exceptions, such^as the TU?TOR language). If only one of the 
features is needed, it can probably be obtained more readily 
or inexpensively elsewhere than the PLATO system. However./ 
pne-of the great virtues of the PLATO system is the fact 
that many useful facilities are available from and- integrated 
into a single computer system. 

Spme.^^^>*mples of these fruitful combinations: 

GomputerTmanagem„ent of CAI and ^off-Hne materials. 
Realistic CBE simulations via graphics, database, and. 

interterminal cdmmurii features and terminal 

peripherals. 

Powerful teaching strategies involving sophisticated 

branching algo^-ithms, drill paradigms, computerized 
diagnosis of student weaknesses, and student control 
of sequencing. 
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On-line conferencing with immediate, (telephone-lyke) or 
delayed^ ( letter-iike) response among large 
populations... 
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CdNTINGENCY PLANNI'NG 

* " Few laws are so invariant and pft-qubted. as Murphy's: 
.*if anything can go wrong., it will." Realizing the wide 
applicability of this law, it is only reasonal5le to construct 
contingency plans for s^pnie of the predictable crises in the 
life; of a CBE site. Below is .a list of likely problems and 
;p](ausible alternatives. 

The system or local site experiences a "down" period longer 
than one class per ibd-ras long as several days. 

Teach conventionally in lecture format using lesson 

authors as instructors; 

Use hardcopy version of CBE lessons prepared before the 
downtime period . 
Postpone training. 

Lesson production goal^s are not met, or substantial loss of 
experienced personnel occurs. 

If avail/ble, hire expensive experienced staff;. 

If time permits, train hew staff. 

Reduce the scope of the pj^oject. 

Extend the term of the project. 

Hire gutside contractors to complete the development. 

The student enrollmeht in target courses becomes so low- that 
evaluation statistics will not be reliable. 

Extend the project to increase the sample size. 

If you are dividing the students, into experimental and 

control groups, make them ail experimental and use^ 

previous classes for control . 

Modify the materials to. teach a high-flow class. 
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The studen.t ehrbllment becomes so Targe that there are too 
many students to schedule ,^ or access_tb terminal time is 
^reduced so that sufficient terminal time is not available. 

Reduce the amount of authoring and the number of 

terminals devoted to authoring. 

Schedule authoring for weekends^and evenings. 

leach part of the students conventionally. 

Allow students to self- schedule* themselves during 

non^prime time. 

i 

There 4s a persistent problem with obtaining sufficient 
computer storage JCS) for ail student lessons. 

ScheduTe classes so that students are likely to .be^ 
sharing a lesson in ECS. Essentially this means 
reducing the self-pace, mode and adopting a more lock-. 
.„ step orientfation . 

If practical, invest the time of programmers to divide 
very large lessons. 

The .funding or ability to .perform the project is suddenly 
terminated. ,* 

Prepare hardcopy versions olT all completed instruction. 

Re-;program the materials for other computers or rewrite 

the materials for other media. 

Although none of the potential alternatives is supplied 
with a caveat, each should be! Note aTso that this list is 
far from^ complete. One of the main purposes of these guide- 
lines is to present examples of situations', problems, and 
alternatives. A more complete: list of contingencies would 
contain options- for problems, with; poor lesson; quality-, ^person 
nel problems, skeptical traditional instructors, etc. the 
manager of each site should prepare his own list, and if 
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necessary, get it approved* Such advanced planning allows 
for thoughtful procedures such as the "backwards implement 
ticri"' discussed in part I. 
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GUIDELINES TOR C0HS7RIJCTING PROJECT TIMELINES 

One of the most important problems endbuntered in 
plainning and managing a CBE site is that of estimating the 
size of the staff and. the; total time needed to complete a 
major lesson development effort. Sites frequently over- 
estimate their production efficiency or fail to take account^, 
of non-production tasks and are thus unable to complete the 
intended' pro ject. For that reason these guidelines^ have 
already recommended that sites allow for ^*soft" failures or 
ways to demonstrate success despite not achieving all 
-original goals. 

Although "soft" failures are. better than "hard" ones^ 
what is nee;ded is a better basis for determining the magni- 
tude of the task and the productivity of the staff. The> 
ability to make accurate forecasts will lower the frequency 
and se.yerity of mid-project adjustments and reduce the 
uncertainties of CBE courseware development. 

The following discussion- assumes that CBE is new to 
nearly all staff members at a site... (After a site's staff 
members have experience^ they can make estimates by extrap- 
platioh?, but during the planning stages of a site, no lessons 
have been written. Yet this is the period when the^best 
estimates are heeded.) 

In- spite of the fact that local situations create issues 
which cannot be answered by generalizations,, we feel that 
we have gleaned some ihsignt which can be used in- project 
planning and management. In order to make the problems and 
constraints as realistic as possible, we have prepared the 
foilowing^ example of how NOT to use estimates of lesson 
production time^ to plan a project. 
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Project History of m h 

the NeverNeverLand project staff read several reports 
which indicatiBd^ that groups similar to theirs had, spent 250 
author hours tc write each hour of lesson material, the MNL 
project wished to create 100 hours of CBE material and calcu- 
lated an author working. 2000 hours per year could develop 
eight lessons. They thus requested and received authoriza- 
tion, to hire 13 staff for two years. They were told^ that if 
the evaluation results were good, the project would he con- 
tinued. All the authors, prpg'rammers , illustrators, and 
technicians were working full time within* three months after 
the project started, and within six months they had learned 
enough of the authoring language to be producing lessons. 

The fact that the last three months of the rproject were 
set aside for evaluation and could hot actually be used for 
development did hot bother the project director because the 
rate of lesson production was faster than the assumed rate. 
During the second quarter of the first year of the project^ 
the average production time was 250 hours per hour; the 
third quarter, it was down to 150 hours per hour! Though 
the fully-operational lesson development period was to be 
only 15 months (24 months minus 6 months arriving and 
..getting trained minus 3 months evaluation), the production 
rate achieved at NNL seemed destined to be twice as fast as 
had been predicted. 

The project director-^was surprised when the fourth 
quarter production rate rose to 175 hours per hour rather 
than continuing to decline ss he thought it would. Feeling 
his authors had become a little too relaxed, he put on the 
pressure to produce. During the fifth quarter, it took an 
average of 150 hours to produce each one hour lesson, but two 
a.uthors disliked the pressure and quit.. With only six months 
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left, hiring untrained staff was out of the question; instead 
bne experienced, but expensive, author was hired to replace 
^the two who had quit.. 

Before the final evaluation, period began, several prob- 
lems reared- The NNL project directo'r^s boss asked to see 
some evaluation data. to show department administrators so 
that the project could continue to be fund^ed.^ The boss 
understo6,d thajt the evaluation period had not begun, but he 
needed some evidence that the CBE system should be kept, and 
he; needed it before the end of the current contract. He, 
would even be happy with lesson validation data, he said. 

Unfortunately, lesson valddation data was in. short 
.supply. The project director admitted tha^t the lessons used 
during the evaluation period would include some which had not 
been validated prior to the evaluation period- (i.e.-, they had 
hot yet been shown to teach effectively), and some lessons 
which were not even written yet but which hopefully would be 
finished during the evaluation period. The difflaulty was 
caused^ by the fact that only three classes a year were sched- 
uled (Fall, Spring, and Summer). Most lessons were revised 
twice before they were validated and^ hence feW lessons 
written during the second year had had enough use to have 
reached* their validation criterion. Most of the validated 
lessons were those written the first year, when production, 
was slow and experience was minimal. 

Lacking sufficient data on which to base a decision, the 
department was forced (rather unhappily) to grant the project 
an extension for only a few months, until the evaluiation 
phase could be completed. The department was willing to con- 
tinue the project, but needed evidence to justify that decis- 
ion to a budget-conscious administration. 

During this extension period, the NNL staff divided 
their time between running the evaluation and finding new 
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jobs in more stable environments; Though the evaluation 
results were seemingly positive, only one class of students 
wais included in> the evaluation sample (because only three 
classes started a year and only one could be run in, the three 
months set aside for evaluation). With such a small sample, 
statistical tests had only limited power to detect differences 
in student performance. Furthermore, some skeptics said that 
summer school students were so atjTpical that the results 
would hot be applicable to ±he students in the other sessions. 

Faced with a depleted staff, positive evaluation results, 
and another long training .period for new staff, the depart- 
.meht funded: CBE support for continued student operations and 
lesson maintenance, .but no more new development. Some 
members of the administration felt that the department had 
been used as a training ground for CBE authors^ who^ left for 
better jobs after being trained . Though they acknowledged 
the student satisfaction and the enhanced performance, they 
discouraged any new lesson development. 

The reader should be advised that although there= is no 
HeverNeverLand site, the details- of this fictional account 
are a composite of events from the case histories of several 
ac:tual srtes; 

Cet us examine the errors and misconceptions of the NNL 
staff in planning their project. 

Post mortem Analy sis 

The story ended in the place where the planning should 
have started: the evaluation. After choosing objectives, 
one should begin immediately to decide how to evaluate the 
project. Using "power analysis" techniques (Cohen,^ 1969), 
one can determine the minimum number of subjects one needs 
during the summative evaluation period to have a reasonable 
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chance of detecting, performance differences. That Informa- 
tion,, combined .with average class sizes, can be used .to deter- 
mine, the length of the evaluation p.eridd. As NNL found out,, 
the need: for evaluation data comes before tjie Xast day of the 
;Pr6ject. Because final evaiuatrohs are usually not completed ' 
before the end of the project, it is necessary to collepr and 
interpret some preliminary data. That can come from the 
formative development or validation data or, if several 
classes of students participate, in the evaluation, perio'd", 
from data from an >early class. ' .^^^^ 

Assume that power analvsis indicates that 100 students 
are needed for the. evaluation i If classes average Ao stu- 
dents and: are taught three times per year, the entire final 
year of the project will be needed to gather summative data; 
If the lessons require two revisions before val-idatibn, threes 
groups of students must use each lesson, thus adding perhaps 
another year to the project*. (Because initial testing; and 
formative evaluation can be- conducted with students from 
other courses in many cases, the one-year validation period 
can often be- shortened. ) 

At the beginning of a project^ staff are learning about 
the language and PBE; lesson development may be slow for ^ to 6 
:months. Using the' assumptions above, one finds nearly 30 
months are needed for training., validation, and evaluation 
alone. Development time must be added to the 30 month sum. 
Unless the validation or evaluation periods can be shortened:, 
such a project must be more than three years in length. 

Projects which attempt to perform summative evaluation 
simultaneously with lesson validation have the following, 
problems. Since the lessons are changing during the evalua- 
tion period, the summative data collected are not comparable. 
Furthermore,, problems with one lesson affect other lessons, 
making detection and location of pedagogical stumbling blocks 
more difficult than if the lessons were ^Validated individually. 
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includes. Most staff who prepare lessons also spend time 
revising lessons, giving demoh*stratioris , using the lessons 
to, teach, reviewing lessons of other authors, constructing 
routers, aiding, the evaluation , and; training new staff. Half 
or more of the time of staff may thus he spent in duties not 
directly related- to lessdn-prdductiori'. That is why the NML 
seemed to be doing so well. A rate of 250 hours may have, 
characterized the entire MfIL project, ef^fort (e.g.,, the'total 
time consumed by each author during; ai>^"stages of the^pro- 
jeat divided by the output of the project). Production of 
first draft lessons generally takes less than that 250 hours 
per hour and certain individual lessons often-can be pro- 
grammed in vfery much less time. A further source of confusion 
in reporting production rates i's deciding which author-support 
time to include. The time of programmers is certainly includ- 
ed,. but should time for roctors , ^secretar ies , illustrators,, 
and' hardware repairmen also be counted? 

Courseware production times can be misleading for 
several reasons. When starting with an inexperienced staff, 
a project director finds that the "learning curve" reduces 
production times quickly. The first, lesson each author 
writes may take 600 hours to produce, the second 300, the 
third 200. Furthermorey each lesson tends to improve in 
quality, the production time seems destined to continue 
decreasing. Instead, three factors tead to , make the produc- 
tion time level off or even increase. 

A major burden for authors mid-way through a project is 
the increasing quantity of data gathering, interpretation, and 
revision activities required. Initially,, an author has only 
a few lessbns to revise; he can spend most of his time creat- 
ing new lessons. There may be only a few periods when there 
are students at an appropriate .-place in their course to use 
his lessons. After a year or so, the author must devote a 
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.consider'table amount of his time to- revising and ^maintaining, 
existing lessons and testing new ones, 

A second^^ factor t,eading to increase courseware produc- 
tion time is the finding that the ^'learning curve" typically, 
shows: "setbacks" (Avner, personal communication) during which 
a new author may drop back to low production rates.. Avner 
discpv,ered that these plateaus usually happened when the 
author was perfecting a new instructional technique. Each 
new -technique added to the repertoire of the new author would 
show a "learning, curve" of production eff icien^y _as it_^ . 
Applied: to mpre and more lessons. Unttl an adequate reper- 
toire was developed for the specific needs of the author ?s 
project, it was quite common to see erratic shifts, of from 20 
to "666^ hours per hour as an Individual author added and 
exploited ea>h' new technique. In areas wher.e a wide range of 
techniques, are required, It took a year or more to deyelop ^a 
repertoire adequate to make fuii use of the capabirities of 
CBE. However, once the repertoire was present, exceedingly 
efficient production of effective material often resulted . 
Taking into consideration the amount of time needed to learn^ 
these skills and the amount of effort often required of an 
author for other duties, it is not realistic to expect that 
many one-hour lessons will be produced^ by a neW author in 17 
hours during ar 3-year project. 

This need for training of users of a medium should not 
be surprizing. Few would expect a group of novices to make 
effective and efficient use of even well-established media 
such as. film or videotape without a substantial period of 
learning and. practice. At the same time, it is easy to be 
misled by the apparent ease with which a novice is able to 
produce first-draft materials in almost any medium (film and 
CBE included) . 

As, a project matures, its authors gain additional re- 
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.s,ppn Sibil i ties. When few lessons have been written, there 
is very little need for routers^, for bookkeeping .abput the 
names and locations of lessons, and for scheduling and 
proctoring classrooms full of students. These activities.^ 
are added, to the tasks of authoring, reviewing less^ons and-: 
preparing Tor the evaluation. These accutfiulatihg duties act 
as third force to slow lesson productipn during project, 
-maturit^y; 

A final problem suggested by the NNL example is that 
estimates oC the number of developer s needed should be adjiiist 
ed upward to allow for non-development staff such as proctors 
i'ilustrators , repairmen ^ etc.. These staff may be needed 
only part time during limited phases of the project.. Their 
skiris« are important , but one cannot expect to r-^place devel- 
opment staff with support staff and :still meet the. same pro- 
ductlon goals. 

A cautionary note about the production rate of 25,0 
hours used in the NNL example is necessary. Several projects 
have, derived or used a value of 2^()r3^^ hours for overall 
production time including training', ^ revision , etc. We* can- 
;riot endorse this or any other "rule of thumb" value because 
of the vast differences between sites and projects. In the 
earliest stages of project planning, it might be reasonable 
to estimate 300 hours per hour of material to approx imate^the 
scope of a. project. After the scope is established, the 
objectives determined, and an evaluation sketched out, one 
shc.uld request several independent QBE consultants to refine 
the plans and provide estimates of project milestones, staff 
needs, and timelines. Providing rules for making such— — 
estimates is beyond the purview of these guidelines. The 
consultants* estimates must be -verified during the first year 
pf tKe project, .observing the caveats, noted in this section 
about the. fickleness of trends. 
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If it is 'not possible to obtiain the aid of a qualified 
CQhsiiltant, yoii should use very conservative estimates. Avner 
Ip.ersonal communication)v indicates that^> based on a study of 
27 new authors, oae could be 90% confident that 75% of a 
group of experienced teachers with good verbal skills (all, 
^were college graduates) would be able to prpduce an hour of 
material during their first year in about il50 hours or less 
of effort. This estimate includes only minimal formative 
^evaluation . A rule^of-thumb. based on these data would thus 
sugges.t that a project with- a one-year lesson development 
phase should allot no more than 5 hours of material to each 
new author durihg^ the year and tnat ii authors^ be prolvided for 
every 15 hours of material to be produced. 
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